Airflow Guides

Topics

Airflow UI
Azure
Basics
Best Practices
Components
Concurrency
Connections
DAGs
Data Quality
Database
Dependencies
Executors
Hooks
Infrastructure
Integrations
Kubernetes
Logging
Machine Learning
Operators
Parallelism
Plugins
Resources
SQL
Secrets
Subdags
Task Groups
Tasks
Templating
Testing
Windows
Workers
XCom
NEW

Scheduling and Timetables in Airflow

Everything you need to know about scheduling your Airflow DAGs.

Airflow Data Quality Checks with SQL Operators

Executing queries in Apache Airflow DAGs to ensure data quality.

Airflow Pools

Using pools to control task parallelism in Airflow.

Integrating Airflow and dbt

Running dbt models in your Airflow DAGs.

Using Airflow with SageMaker

Methods for orchestrating SageMaker machine learning pipelines with Airflow.

Executing Notebooks with Airflow

Methods for orchestrating commonly used notebooks with Airflow.

Cross-DAG Dependencies

How to implement dependencies between your Airflow DAGs.

Testing Airflow DAGs

How to apply test-driven development practices to your Airflow DAGs.

Using Task Groups in Airflow

Using Task Groups to build modular workflows in Airflow.

Custom XCom Backends

Creating a custom XCom backend with Airflow 2.0.

Passing Data Between Airflow Tasks

Methods for sharing metadata and information between tasks in your Apache Airflow DAGs.

Deploying Kedro Pipelines to Apache Airflow

How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.

Orchestrating Databricks Jobs with Airflow

Orchestrating Databricks Jobs from your Apache Airflow DAGs.

Executing Azure Data Factory Pipelines with Airflow

Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.

Executing Azure Data Explorer Queries with Airflow

Executing Azure Data Explorer queries from your Apache Airflow DAGs.

Orchestrating Azure Container Instances with Airflow

Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.

Get Started with Apache Airflow 2.0

Test Apache Airflow 2.0 on your local machine with the Astronomer CLI.

Using Airflow to Execute SQL

Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.

Integrating Airflow and Great Expectations

Using the Great Expectations provider natively in your Airflow DAGs.

Understanding the Airflow Metadata Database

An structural walkthrough of Apache Airflow's metadata database, with a full ERD.

Executing Talend Jobs with Airflow

Triggering remote jobs in Talend from your Apache Airflow DAGs.

Integrating Airflow and Hashicorp Vault

Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.

Importing Custom Hooks & Operators

How to correctly import custom hooks and operators

Scaling Out Airflow

How to scale out Airflow workers and the settings needed to maximize parallelism

Airflow Executors Explained

A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.

Logging in Airflow

Demystifying Airflow's logging configuration.

Introduction to Kubernetes

High-level overview of introductory concepts in Kubernetes.

Best Practices Calling AWS Lambda from Airflow

A few tips, guidelines, and best practices for calling Lambda from Airflow

Using Kerberos in Apache Airflow

How to use Kerberos and Kerberized hooks in Airflow

Running scripts using the BashOperator

Learn and troubleshoot how to run shell scripts using the Bash Operator in Airflow

KubernetesPodOperator on Astronomer

Use the KubernetesPodOperator on Astronomer

Using SubDAGs in Airflow

Using SubDAGs to build modular workflows in Airflow.

Templating in Airflow

How to leverage the power of Jinja templating when writing your DAGs.

Branching in Airflow

Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow

Airflow's Components

How all of Apache Airflow's components fit together.

Useful SQL queries for Apache Airflow

A home for SQL queries that we frequently run on our Airflow postgres database.

The Airflow UI

A high-level overview of the Airflow UI

Running Airflow on Windows 10 & WSL

How to spin up Airflow on your Windows system.

Managing your Connections in Apache Airflow

An overview of how connections work in the Airflow UI.

DAG Writing Best Practices in Apache Airflow

How to create effective, clean, and functional DAGs.

Intro to Apache Airflow DAGs

What are DAGs and how they are constructed in Apache Airflow?

Dynamically Generating DAGs in Airflow

Using a base DAG template to create multiple DAGs.

Editing Task and DAG Metadata

What are DAGs and how they are constructed in Apache Airflow?

Error Notifications in Airflow

Methods for managing notifications in your Airflow DAGs.

From Operators to DagRuns

From Operators to DagRuns

Introduction to Apache Airflow

Everything you need to know to get started with Apache Airflow.

Managing Airflow Code

Guidelines for Working with Multiple Airflow Projects

Managing Dependencies in Apache Airflow

An overview of dependencies and triggers in Airflow.

Scheduling Tasks in Airflow

Overview of the Airflow scheduler

Using Apache Airflow Plugins

A crash-course in using Airflow Plugins.

Hooks 101

An introduction to Hooks in Apache Airflow.

Sensors 101

An introduction to Sensors in Apache Airflow.

Operators 101

An introduction to Operators in Apache Airflow.

Never miss an update from us.