Astronomer Blog

The latest insights from our team of experts

September 22, 2023

Ask Astro: An open source LLM Application with Apache Airflow

Michael Gregory Staff Ecosystem Engineer

A reference implementation and community dialogue for building production-grade day-2 operations for LLM applications

September 8, 2023

Change Data Capture with Airflow - Part 2

We look at some examples to implement CDC in Airflow using a legacy data pipeline with batch processing, and look at advanced use cases for near-real-time processing.

August 30, 2023

Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow, Part 1

A detailed guide to building an ETL workflow in Databricks Workflows and Apache Airflow

July 26, 2023

Introducing Cosmos 1.0: the best way to run dbt Core in Airflow

Turn your dbt projects into entire Airflow DAGs with less than 10 lines of code.

July 25, 2023

6 Lessons Learned in Building Astronomer’s Developer Documentation

The Astronomer Approach to Clear and Effective Technical Documentation

July 20, 2023

Expert tips for monitoring the health and SLAs of your Apache Airflow DAGs

Metrics in the form of tables, charts, or graphs are useful for monitoring the health of your Airflow system and maintaining the SLA of your data pipelines. They provide a quick and easy way to identify issues and take corrective actions. While Airflow's built-in UI is a good starting point along with notifications, using Airflow’s REST API to extract metadata about your DAGs and tasks provides advanced observability options with an added overhead of cost and maintenance.

July 18, 2023

Advantages of Hosted Airflow for Your ETL Workflows

Maximize ETL efficiency with hosted Apache Airflow on Astro, not self-hosting open-source. Benefit from simplified infrastructure management, scalable elasticity, and dedicated support for your workloads.

July 13, 2023

Best Practices for Building an Airflow Service (Part 1)

This guide covers best practices for everything from choosing the right Airflow deployment model to configuring your DAGs for optimal performance.

July 12, 2023

Astronomer and Fivetran Partner to Release Production-Grade ELT Airflow Provider

The new provider will make it easier for organizations to use Airflow to automate and manage their Fivetran pipelines.

June 26, 2023

Astronomer and Snowflake: Unleash the Power of Snowpark Container Services and Apache Airflow

We are excited to announce the partnership between Astronomer and Snowflake, aimed at revolutionizing the way organizations leverage data and execute their data pipelines.

June 22, 2023

Three ways to use Airflow with MotherDuck and DuckDB

Use Apache Airflow with DuckDB and MotherDuck in three different ways. Access the DuckDB Python package directly, leverage the DuckDB Airflow provider, and use DuckDB with the Astro Python SDK.

June 13, 2023

The Astro Cloud IDE: from Python and SQL to nearly 1,000 Airflow operators

Use ~1,000 open-source Airflow operators and define your own custom operators in the Astro Cloud IDE with the newly released cell type functionality

Astronomer Blog

Try Astro Today