Astronomer Blog

The latest insights from our team of experts

September 22, 2023

Ask Astro: An open source LLM Application with Apache Airflow

Michael Gregory Staff Ecosystem Engineer

A reference implementation and community dialogue for building production-grade day-2 operations for LLM applications

September 21, 2023

Day 2 Operations for LLMs with Apache Airflow: Going Beyond the Prototype

We discuss the challenges of LLM application development beyond the prototype and why Apache Airflow was highlighted for its ability to enable day-2 operations for LLM applications.

September 8, 2023

Change Data Capture with Airflow - Part 2

We look at some examples to implement CDC in Airflow using a legacy data pipeline with batch processing, and look at advanced use cases for near-real-time processing.

August 30, 2023

Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow, Part 1

A detailed guide to building an ETL workflow in Databricks Workflows and Apache Airflow

August 24, 2023

Debugging Airflow Made Easy: 3 Key Steps to Debug your Airflow DAGs

In this post, we’ll walk through some high level methods for easily debugging your DAGs

August 23, 2023

Orchestrating Machine Learning Pipelines with Airflow

In the realm of machine learning, managing workflows efficiently is paramount. One tool that has emerged as a game-changer in this space is Apache Airflow.

August 22, 2023

Change Data Capture in Apache Airflow - Part 1

A complete guide to handling CDC in Airflow with ease.

August 18, 2023

Introducing Airflow 2.7

The latest minor release includes several new features, such as automatic setup/teardown of tasks, built-in OpenLineage support, cluster activity view, fail-stop functionality, and more.

August 15, 2023

Leveraging Apache Airflow and Kubernetes for Data Processing

Get insights on how to use Apache Airflow and the Kubernetes Executor for data processing, along with proven best practices and tips for scaling your workloads.

August 10, 2023

Test Airflow Upgrades with the Astro CLI

The Local Upgrade Test command in the Astro CLI eliminates upgrade pains and ensures safe upgrades, allowing users to confidently identify and resolve compatibility issues, and DAG import errors.

August 9, 2023

Enhanced Astro Workspace Roles for more granular permissions

In this blog post, we will dive into the details of the Astro’s Role-Based Access Control (RBAC) and new Workspace Role updates, and explore improvements to popular use cases of Astro.

August 1, 2023

ETL in Airflow: A Comprehensive Guide to Efficient Data Pipelines

This guide covers everything you need to know about creating efficient data pipelines with Apache Airflow. Get tips to speed up your pipelines, improve their reliability, and make them easier to manage.

Astronomer Blog

Try Astro Today