Airflow Guides


Airflow UI
Best Practices
Data Quality
Machine Learning
Task Groups

Astro for ETL

Using the `astro` library to implement ETL use cases in Airflow.

Introduction to Airflow Decorators

An overview of Airflow decorators and how they can improve the DAG authoring experience.

Deferrable Operators

How to implement deferrable operators to save cost and resources with Airflow.

Debugging DAGs

A beginner's guide to figuring out what's going wrong with your Airflow DAGs

Rerunning Airflow DAGs

How to use catchup, backfill, and cleared task instances in Airflow.

Scheduling and Timetables in Airflow

Everything you need to know about scheduling your Airflow DAGs.

Airflow Data Quality Checks with SQL Operators

Executing queries in Apache Airflow DAGs to ensure data quality.

Airflow Pools

Using pools to control task parallelism in Airflow.

Integrating Airflow and dbt

Running dbt models in your Airflow DAGs.

Using Airflow with SageMaker

Methods for orchestrating SageMaker machine learning pipelines with Airflow.

Executing Notebooks with Airflow

Methods for orchestrating commonly used notebooks with Airflow.

Cross-DAG Dependencies

How to implement dependencies between your Airflow DAGs.

Testing Airflow DAGs

How to apply test-driven development practices to your Airflow DAGs.

Using Task Groups in Airflow

Using Task Groups to build modular workflows in Airflow.

Custom XCom Backends

Creating a custom XCom backend with Airflow 2.0.

Passing Data Between Airflow Tasks

Methods for sharing metadata and information between tasks in your Apache Airflow DAGs, including XCom.

Deploying Kedro Pipelines to Apache Airflow

How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.

Orchestrating Databricks Jobs with Airflow

Orchestrating Databricks Jobs from your Apache Airflow DAGs.

Executing Azure Data Factory Pipelines with Airflow

Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.

Executing Azure Data Explorer Queries with Airflow

Executing Azure Data Explorer queries from your Apache Airflow DAGs.

Orchestrating Azure Container Instances with Airflow

Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.

Get Started with Apache Airflow 2.0

Test Apache Airflow 2.0 on your local machine with the Astronomer CLI.

Using Airflow to Execute SQL

Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.

Integrating Airflow and Great Expectations

Using the Great Expectations provider natively in your Airflow DAGs.

Understanding the Airflow Metadata Database

An structural walkthrough of Apache Airflow's metadata database, with a full ERD.

Executing Talend Jobs with Airflow

Triggering remote jobs in Talend from your Apache Airflow DAGs.

Integrating Airflow and Hashicorp Vault

Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.

Importing Custom Hooks & Operators

How to correctly import custom hooks and operators.

Scaling Out Airflow

How to scale out Airflow workers and the settings needed to maximize parallelism

Airflow Executors Explained

A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.

Logging in Airflow

Demystifying Airflow's logging configuration.

Introduction to Kubernetes

High-level overview of introductory concepts in Kubernetes.

Best Practices Calling AWS Lambda from Airflow

A few tips, guidelines, and best practices for calling Lambda from Airflow

Using Kerberos in Apache Airflow

How to use Kerberos and Kerberized hooks in Airflow

KubernetesPodOperator on Astronomer

Use the KubernetesPodOperator on Astronomer

Running scripts using the BashOperator

Learn and troubleshoot how to run shell scripts using the Bash Operator in Airflow

Using SubDAGs in Airflow

Using SubDAGs to build modular workflows in Airflow.

Templating in Airflow

How to leverage the power of Jinja templating when writing your DAGs.

Branching in Airflow

Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow

Airflow's Components

How all of Apache Airflow's components fit together.

Useful SQL queries for Apache Airflow

A home for SQL queries that we frequently run on our Airflow postgres database.

The Airflow UI

A high-level overview of the Airflow UI

Running Airflow on Windows 10 & WSL

How to spin up Airflow on your Windows system.

Managing your Connections in Apache Airflow

An overview of how connections work in the Airflow UI.

DAG Writing Best Practices in Apache Airflow

How to create effective, clean, and functional DAGs.

Intro to Apache Airflow DAGs

What are DAGs and how they are constructed in Apache Airflow?

Dynamically Generating DAGs in Airflow

Using a base DAG template to create multiple DAGs.

Editing Task and DAG Metadata

What are DAGs and how they are constructed in Apache Airflow?

Error Notifications in Airflow

Methods for managing notifications in your Airflow DAGs.

From Operators to DagRuns

From Operators to DagRuns

Introduction to Apache Airflow

Everything you need to know to get started with Apache Airflow.

Managing Dependencies in Apache Airflow

An overview of dependencies and triggers in Airflow.

Managing Airflow Code

Guidelines for Working with Multiple Airflow Projects

Using Apache Airflow Plugins

A crash-course in using Airflow Plugins.

Hooks 101

An introduction to Hooks in Apache Airflow.

Sensors 101

An introduction to Sensors in Apache Airflow.

Operators 101

An introduction to Operators in Apache Airflow.