Airflow in Action: DataOps Insights from 2,200 Pipelines at Instacart
- M Matthew Keep
Discover how Instacart scaled Apache Airflow to handle 2,200 pipelines and 16 million tasks monthly with custom tools, IaC, and centralized management.
Astronomer Blog
The latest insights from our team of Apache Airflow® experts.
Airflow in Action: ML and Data Engineering Insights from the Analytics-Obsessed World of MLB
Explore how the Texas Rangers and Philadelphia Phillies use Airflow to revolutionize data orchestration and ML, driving faster insights and a competitive edge.
Data Observability 101: An Introduction to the Most Critical Features of Modern Data Observability
When data is your product, your pipelines are supply chains.
Apache Airflow® TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs
Explore the TaskFlow API and traditional operators and find out how to combine them for dynamic, efficient DAGs.
Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider
See how the new Tecton Airflow Provider can make your feature pipeline orchestration within Apache Airflow® more efficient.
Ask Astro: Operationalizing Data Ingest for Retrieval Augmented Generation with LLMs, Part 3
In part 3 of Ask Astro series, uncover key considerations for data ingestion in Retrieval Augmented Generation with LLMs. Find out how to select the ideal vector store, model, schema design, and chunking strategy for your project.
Using Astronomer’s new Cosmos to deploy dbt pipelines onto Snowflake
The state of deploying pipelines with dbt has changed considerably in the last few months. Over the last few weeks, I was working with Astronomer to test out their new tool, Cosmos, to deploy dbt workflows onto Snowflake.
Databricks vs. Airflow From a Management Perspective, Part 2
Databricks vs Airflow from a production management perspective. Explore the differences in setup, monitoring, integrations, scalability & customization.
ML for Customer Analytics with Airflow, Snowpark, and Weaviate
An example project showing how to use Apache Airflow® to orchestrate a machine learning pipeline with the Snowpark provider and Snowpark ML.
Migrate Python Jobs to Airflow in 4 Simple Steps
Learn how easy it is to migrate Python scripts to Airflow DAGs, streamline orchestration, and leverage Airflow features to boost job efficiency.
3 Key Takeaways from Airflow Summit 2023
Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.
Ask Astro: An open source LLM Application with Apache Airflow®, Part 2
Build an LLM-powered chatbot with Airflow! Learn how to leverage domain-specific knowledge to create intelligent applications like "Ask Astro." Astro LLM meets Apache Airflow.
