Astronomer Blog

The latest insights from our team of experts

September 22, 2023

Ask Astro: An open source LLM Application with Apache Airflow

Michael Gregory Staff Ecosystem Engineer

A reference implementation and community dialogue for building production-grade day-2 operations for LLM applications

September 8, 2023

Change Data Capture with Airflow - Part 2

We look at some examples to implement CDC in Airflow using a legacy data pipeline with batch processing, and look at advanced use cases for near-real-time processing.

August 30, 2023

Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow, Part 1

A detailed guide to building an ETL workflow in Databricks Workflows and Apache Airflow

January 5, 2021

Building a Scalable Analytics Architecture With Airflow and dbt: Part 2

Now that we have these DAGs running locally and built from our dbt `manifest.json` file, the natural next step is to evaluate how these should look in a production context.

December 22, 2020

Building a Scalable Analytics Architecture With Airflow and dbt

Implementing an ideal development experience at the intersection of two popular open-source tools, written in collaboration with our friends at Updater.

December 17, 2020

The Airflow 2.0 Scheduler

A technical deep-dive into Apache Airflow's refactored Scheduler, now significantly faster and ready for scale.

November 23, 2020

A Great Expectations Provider for Apache Airflow

We're pleased to announce an official integration that allows users to leverage Great Expectations natively in their DAGs.

October 29, 2020

Introducing Airflow 2.0

A breakdown of the major features incorporated in Apache Airflow 2.0, including a refactored, highly-available Scheduler, over 30 UI/UX improvements, a new REST API and much more.

March 17, 2020

Introducing KEDA for Airflow

Using KEDA (Kubernetes Event-Driven Autoscaler), we've developed a robust method to scale Apache Airflow workers to be faster and more versatile than any previous architecture.