Astronomer and Snowflake: Unleash the Power of Snowpark Container Services and Apache Airflow
We are excited to announce the partnership between Astronomer and Snowflake, aimed at revolutionizing the way organizations leverage data and execute their data pipelines. As a trusted leader in the workflow management space, Astronomer has been selected as a launch partner for Snowflake’s highly anticipated Snowpark Container Services launch to help bring code closer to data for the entire Apache Airflow community.
Snowflake and Apache Airflow
Apache Airflow is the de facto open source platform for writing data pipelines as code, and a standard part of the modern data stack for companies building with Snowflake. With a massive community of battle-tested integrations, Airflow allows users to write pipelines as directed acyclic graphs (DAGs) to orchestrate how data moves into Snowflake, as well as to execute transformations, data quality checks and sensors that trigger data preparation tasks with Snowflake’s powerful SQL execution engine. Airflow operators significantly simplify this process and the handling of intermediate data sets via the cross-communication (XCOM) system.
Snowpark is the set of libraries and runtimes that securely enable developers to deploy and process non-SQL code in Snowflake.
We’re excited to announce an updated Snowpark provider for Airflow (Apache 2.0 license) with three new features we’ve developed with Snowflake for the Airflow community:
- Snowpark Operators (in public preview), which bring the power of Airflow to Snowpark Python tasks.
- A new XCOM backend for Snowflake to ensure that intermediate data passed between tasks stays in Snowflake regardless of data size and complexity.
- Snowpark Container Services Operator (in private preview) allows data teams to orchestrate tasks in any language and any workflow from data engineering to machine learning (with GPUs) and application development.
The full suite of Airflow integrations, along with examples of using them with best practices, can be found on the Astronomer Registry. And, check out our recent webinar on how to use Snowpark with Airflow.
More on Snowpark Container Services
Snowpark Container Services, launching in private preview, extends Snowflake’s processing engine to provide developers the flexibility to register and deploy containerized software and applications in Snowflake-managed infrastructure. Among other things, Snowpark Container Services empowers execution of containers directly within customers’ Snowflake account. This helps teams be more self-sufficient, as it simplifies dependency management and the provisioning and maintenance of compute infrastructure while reducing cost and complexity of data egress. By eliminating the need to copy and move data out of Snowflake, organizations can ensure robust security and data governance throughout the entire data lifecycle.
“No Knobs” with Astro and Snowflake
Astro, a fully-managed service for Apache Airflow from Astronomer, simplifies the deployment and management of Apache Airflow, making it an ideal solution for data teams who prioritize developer productivity. By leveraging Astro’s fully-managed Airflow service in conjunction with the full array of Snowflake offerings, organizations can focus on building robust data pipelines without worrying about the complexities of managing Airflow. The Astro Cloud IDE and Snowpark Container Services make it possible for users to build pipelines visually without even thinking about infrastructure.
Just as Snowflake’s commitment to a “no knobs” engineering philosophy enables teams to build data platforms in minutes, Astro’s fully-managed cloud service empowers teams to build and scale Apache Airflow services in minutes without the undifferentiated heavy-lifting of managing compute platforms for their data pipelines. Leveraging Astro and Snowflake together lets high-velocity data teams focus on creating business value without having to worry about security, reliability, or scalability.
Discover the power of Astro. Build and optimize your data pipelines without the hassle of managing Airflow. Get started for free with a 14-day trial!
