Airflow Guides

Topics

Airflow UI
Basics
Best Practices
Components
Concurrency
Connections
DAGs
Database
Dependencies
Executors
Hooks
Infrastructure
Integrations
Kubernetes
Logging
Operators
Parallelism
Plugins
Resources
SQL
Secrets
Subdags
Tasks
Templating
Windows
Workers
XCom

Integrating Airflow and Great Expectations

Using the Great Expectations provider natively in your Airflow DAGs.

Executing Talend Jobs with Airflow

Triggering remote jobs in Talend from your Apache Airflow DAGs.

Integrating Airflow and Hashicorp Vault

Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.

Importing Custom Hooks & Operators

How to correctly import custom hooks and operators

Scaling out Airflow

How to scale out Airflow workers and the settings needed to maximize parallelism

Airflow Executors Explained

A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.

Airflow High-Availability

Best practice for running a high-availability Airflow cluster

Logging in Airflow

Demystifying Airflow's logging configuration.

Astronomer vs. Google Cloud Composer

A high-level comparison of Astronomer and Google Cloud Composer

Introduction to Kubernetes

High-level overview of introductory concepts in Kubernetes.

Best Practices Calling AWS Lambda and Cloud Functions from Airflow

A few tips, guidelines, and best practices for calling Lambda and Cloud functions from Airflow

Using Kerberos in Apache Airflow

How to use Kerberos and Kerberized hooks in Airflow

KubernetesPodOperator on Astronomer

Use the KubernetesPodOperator on Astronomer

Running scripts using the BashOperator

Learn and troubleshoot how to run shell scripts using the Bash Operator in Airflow

Airflow Resources

A collection of blog posts, decks, guides, and use cases written by and for the Airflow community.

Hubspot API to Redshift with Airflow

Use Airflow to ingest CRM data from the Hubspot API into Redshift

Marketo API to Redshift with Airflow

Use Airflow to ingest data from the Marketo API to Redshift

MongoDB to Redshift with Apache Airflow

Use Apache Airflow to ingest data from MongoDB to Redshift

Salesforce API to Redshift with Apache Airflow

Use Airflow to ingest CRM data from the Salesforce API to Redshift

Using SubDAGs in Airflow

Using SubDAGs to build modular workflows in Airflow.

Templating and Macros in Airflow

How to leverage the power of Jinja Templating when writing your DAGs.

Trigger DAGs in Airflow

How to use DAGs to trigger secondary DAG kickoffs in Airflow.

Airflow's Components

How all of Apache Airflow's components fit together.

Branching in Airflow

Use Apache Airflow's BranchOperator and ShortCircuitOperator to execute conditional branches in your workflow

Using Airflow Datastores

Use Apache Airflow's internal datastores to build more powerful DAGs

Useful SQL queries for Apache Airflow

A home for SQL queries that we frequently run on our Airflow postgres database.

The Airflow UI

A high-level overview of the Airflow UI

Airflow vs. AWS Glue

How Airflow differs from Amazon's data pipeline orchestration tool, AWS Glue.

Airflow vs. Luigi

How Airflow differs from Luigi.

Airflow vs. Oozie

How Airflow differs from Oozie.

Running Airflow on Windows 10 & WSL

How to spin up Airflow on your Windows system.

Managing your Connections in Apache Airflow

An overview of how connections work in the Airflow UI.

DAG Writing Best Practices in Apache Airflow

How to create effective, clean, and functional DAGs.

Intro to Apache Airflow DAGs

What are DAGs and how they are constructed in Apache Airflow?

Dynamically Generating DAGs in Airflow

Using a base DAG template to create multiple DAGs.

Editing Task and DAG Metadata

What are DAGs and how they are constructed in Apache Airflow?

Error Notifications in Airflow

Managing Error Notifications

Facebook Ads API to Redshift with Airflow

Use Airflow to ingest ad data from the Facebook Ads API into Redshift

From Operators to DagRuns

From Operators to DagRuns

Github API to Redshift with Apache Airflow

Use Airflow to ingest data from the Github API into Redshift

Google Analytics API to Redshift with Airflow

Use Airflow to ingest data from the Google Analytics API to Redshift

IMAP to Redshift with Airflow

Use Airflow to ingest data from an IMAP server to Redshift

Introduction to Apache Airflow

Everything you need to know to get started with Apache Airflow.

Intro to Kubectl

Getting setup and basic commands on kubectl

Managing Airflow Code

Guidelines for Working with multiple Airflow Projects

Managing Dependencies in Apache Airflow

An overview of dependencies and triggers in Airflow.

Using Apache Airflow Plugins

A crash-course in using Airflow Plugins.

Hooks

What is a Hook

Scheduling Tasks in Airflow

Overview of the Airflow scheduler

What is a Sensor

What is a Sensor?

What is an Operator

What is an Operator

Never miss an update from us.

We use cookies to ensure you get the best experience on our website.