Astronomer Blog

The latest insights from our team of experts

September 22, 2023

Ask Astro: An open source LLM Application with Apache Airflow

Michael Gregory Staff Ecosystem Engineer

A reference implementation and community dialogue for building production-grade day-2 operations for LLM applications

September 8, 2023

Change Data Capture with Airflow - Part 2

We look at some examples to implement CDC in Airflow using a legacy data pipeline with batch processing, and look at advanced use cases for near-real-time processing.

August 30, 2023

Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow, Part 1

A detailed guide to building an ETL workflow in Databricks Workflows and Apache Airflow

December 5, 2022

Introducing the Astro Cloud IDE

The Astro Cloud IDE is a notebook-inspired tool for writing data pipelines — no Airflow knowledge required.

December 2, 2022

What’s New in Apache Airflow 2.5

Airflow’s third significant release of 2022 introduces a redesigned DAG test feature, task annotations, data-dependent scheduling enhancements, and more.

November 28, 2022

A Short History of DAG Writing

The needs of the data team have evolved since Apache Airflow was open-sourced by Airbnb in 2015, pushing Airflow itself to evolve. We’re excited about how far we’ve come, and have some big ideas about where Airflow and orchestration can go.

November 17, 2022

Best Practices for Secure Network Connectivity and Authentication in Astro

Get the most out of Astro by incorporating solid connectivity and authentication practices from the start.

November 11, 2022

3 Ways to Extract Data Lineage with Airflow

Learn how to use Airflow’s operators, custom extractors, and inlet/outlet arguments to send lineage to your data observability tool.

November 10, 2022

How Astro’s Data Graph Helps Data Engineers Run and Fix Their Pipelines

Learn how Astro can help you understand, communicate, and solve pipeline problems.

November 3, 2022

OpenLineage: Where It Came from and What Comes Next

Data lineage is having a moment. Julien Le Dem, Co-Founder of OpenLineage and Chief Architect at Astronomer, looks at the drivers behind a notable increase in lineage adoption.

October 27, 2022

How to Keep Data Quality in Check with Airflow

The Airflow-driven data quality checks that we use at Astronomer ensure bad data is found and fixed quickly.

October 18, 2022

What’s New in Astro Python SDK 1.1: Data-Driven Scheduling, Dynamic Tasks, and Redshift Support

Learn what the new upgraded Astro Python SDK 1.1 offers to Airflow users.

Astronomer Blog

Try Astro Today