Apache Airflow Apache Airflow Guides

New

Datasets and Data-Aware Scheduling in Airflow

Using datasets to implement DAG dependencies and scheduling in Airflow.

  • DAGs
  • Dependencies
  • Scheduling
New

Soda Core and Airflow

Using Soda Core to implement data quality checks in Airflow DAGs.

  • Soda
  • Data Quality
New

Data quality and Airflow

Check the quality of your data using Airflow.

  • SQL
  • Great Expectations
  • Data quality
New

Orchestrating Snowflake Queries with Airflow

How to use Airflow to get enhanced observability and compute savings while orchestrating your Snowflake jobs.

  • Integrations
  • ETL
  • Database
New

Dynamic Tasks in Airflow

How to dynamically create tasks at runtime in your Airflow DAGs.

  • Tasks
New

OpenLineage and Airflow

Using OpenLineage and Marquez to get lineage data from your Airflow DAGs.

  • Lineage
New

Orchestrating Redshift Operations from Airflow

Setting up a connection to Redshift and using available Redshift modules.

  • Database
  • SQL
  • DAGs
  • Integrations
  • AWS
New

The Astro Python SDK for ETL

Using the Astro Python SDK to implement ETL use cases in Airflow.

  • Astro
  • ETL
  • SQL
New

Introduction to Airflow Decorators

An overview of Airflow decorators and how they can improve the DAG authoring experience.

  • DAGs
  • Basics
New

Deferrable Operators

How to implement deferrable operators to save cost and resources with Airflow.

  • Operators
  • Concurrency
  • Resources
  • Sensors
  • Workers
New

Debugging DAGs

A beginner's guide to figuring out what's going wrong with your Airflow DAGs

  • DAGs
  • Basics
New

Rerunning Airflow DAGs

How to use catchup, backfill, and cleared task instances in Airflow.

  • DAGs
New

Scheduling and Timetables in Airflow

Everything you need to know about scheduling your Airflow DAGs.

  • DAGs
New

Airflow Data Quality Checks with SQL Operators

Executing queries in Apache Airflow DAGs to ensure data quality.

  • Database
  • SQL
  • DAGs
  • Data Quality
New

Airflow Pools

Using pools to control task parallelism in Airflow.

  • Parallelism
  • Tasks
New

Integrating Airflow and dbt

Running dbt models in your Airflow DAGs.

  • DAGs
  • Integrations
New

Using Airflow with SageMaker

Methods for orchestrating SageMaker machine learning pipelines with Airflow.

  • DAGs
  • Integrations
  • Machine Learning
New

Executing Notebooks with Airflow

Methods for orchestrating commonly used notebooks with Airflow.

  • DAGs
  • Integrations
  • Machine Learning
New

Cross-DAG Dependencies

How to implement dependencies between your Airflow DAGs.

  • DAGs
  • Subdags
  • Dependencies
  • Sensors
  • Dataset
New

Testing Airflow DAGs

How to apply test-driven development practices to your Airflow DAGs.

  • DAGs
  • Best Practices
  • Testing
New

Using Task Groups in Airflow

Using Task Groups to build modular workflows in Airflow.

  • DAGs
  • Subdags
  • Task Groups
  • Best Practices
New

Custom XCom Backends

Creating a custom XCom backend with Airflow 2.0.

  • Plugins
  • XCom
New

Passing Data Between Airflow Tasks

Methods for sharing metadata and information between tasks in your Apache Airflow DAGs, including XCom.

  • DAGs
  • XCom
  • Tasks
  • Dependencies
New

Deploying Kedro Pipelines to Apache Airflow

How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.

  • Plugins
  • Integrations
New

Orchestrating Databricks Jobs with Airflow

Orchestrating Databricks Jobs from your Apache Airflow DAGs.

  • Integrations
  • DAGs
New

Executing Azure Data Factory Pipelines with Airflow

Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.

  • Integrations
  • Azure
New

Executing Azure Data Explorer Queries with Airflow

Executing Azure Data Explorer queries from your Apache Airflow DAGs.

  • Integrations
  • Azure
  • DAGs
New

Orchestrating Azure Container Instances with Airflow

Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.

  • Integrations
  • Azure
  • DAGs
New

Get Started with Apache Airflow 2.0

Test Apache Airflow 2.0 on your local machine with the Astro CLI.

  • Resources
  • Basics
New

Using Airflow to Execute SQL

Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.

  • Database
  • SQL
  • DAGs
New

Integrating Airflow and Great Expectations

Using the Great Expectations provider natively in your Airflow DAGs.

  • DAGs
  • Integrations
New

Understanding the Airflow Metadata Database

An structural walkthrough of Apache Airflow's metadata database, with a full ERD.

  • Database
  • SQL
  • Components
New

Executing Talend Jobs with Airflow

Triggering remote jobs in Talend from your Apache Airflow DAGs.

  • Integrations
New

Integrating Airflow and Hashicorp Vault

Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.

  • DAGs
  • Secrets
  • Integrations
New

Importing Custom Hooks & Operators

How to correctly import custom hooks and operators.

  • Hooks
  • Operators
  • Plugins
  • Basics
New

Scaling Out Airflow

How to tune your Airflow environment so it scales with your DAGs.

  • Workers
  • Concurrency
  • Parallelism
  • DAGs
New

Airflow Executors Explained

A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.

  • Executors
  • Basics
  • Kubernetes
  • Concurrency
  • Parallelism
New

Logging in Airflow

Demystifying Airflow's logging configuration.

  • Logging
  • Best Practices
  • Basics
New

Best Practices Calling AWS Lambda from Airflow

A few tips, guidelines, and best practices for calling Lambda from Airflow

  • Best Practices
  • Integrations
New

Using the KubernetesPodOperator

Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods

  • Kubernetes
  • Operators
New

Using the BashOperator

Use the BashOperator in Airflow to execute bash commands and scripts

  • DAGs
  • Operators
New

Using SubDAGs in Airflow

Using SubDAGs to build modular workflows in Airflow.

  • DAGs
  • Subdags
New

Templating in Airflow

How to leverage the power of Jinja templating when writing your DAGs.

  • Templating
  • Best Practices
  • Basics
New

Branching in Airflow

Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow

  • DAGs
  • Operators
  • Basics
  • Tasks
New

Airflow's Components

Learn about the core components of Apache Airflow's infrastructure.

  • Components
  • Executors
  • Database
  • Basics
New

The Airflow UI

A high-level overview of the Airflow UI

  • DAGs
  • Airflow UI
  • Basics
  • XCom
  • Tasks
  • Connections
New

Running Airflow on Windows 10 & WSL

How to spin up Airflow on your Windows system.

  • Windows
New

Managing your Connections in Apache Airflow

An overview of how connections work in the Airflow UI.

  • Connections
  • Basics
  • Hooks
  • Operators
New

DAG Writing Best Practices in Apache Airflow

How to create effective, clean, and functional DAGs.

  • DAGs
  • Best Practices
  • Basics
  • Templating
  • Tasks
New

Introduction to Airflow DAGs

How to write your first DAG in Apache Airflow

  • Airflow UI
  • DAGs
  • Basics
New

Dynamically Generating DAGs in Airflow

Using a base DAG template to create multiple DAGs.

  • DAGs
  • Best Practices
New

Error Notifications in Airflow

Methods for managing notifications in your Airflow DAGs.

  • DAGs
  • Integrations
  • Operators
New

Introduction to Apache Airflow

Everything you need to know to get started with Apache Airflow.

  • Basics
  • Start
New

Managing Airflow Code

Guidelines for Working with Multiple Airflow Projects

  • DAGs
  • Best Practices
  • Basics
New

Managing Dependencies in Apache Airflow

An overview of dependencies and triggers in Airflow.

  • Best Practices
  • Dependencies
  • Basics
New

Using Apache Airflow Plugins

A crash-course in using Airflow Plugins.

  • Best Practices
  • Plugins
  • Basics
New

Hooks 101

An introduction to Hooks in Apache Airflow.

  • Hooks
  • Operators
  • Tasks
  • Basics
New

Sensors 101

An introduction to Sensors in Apache Airflow.

  • Operators
  • Tasks
  • Basics
  • Sensors
New

Operators 101

An introduction to Operators in Apache Airflow.

  • Operators
  • Tasks
  • Basics