WEBINAR

Using Airflow with Azure Data Factory

Recorded On August 10, 2021

  • Josh Fell
  • Benji Lampel

Welcome to the recap of a webinar on using Airflow with Azure Data Factory!

During the webinar we covered:

  1. What is ADF?
  2. What does ADF do?
  3. Adding Airflow to Azure Data Factory
  4. Demo
  5. Q&A

1. What is Azure Data Factory?

airflow-adf1

Long story short, ADF is an orchestration tool for data pipelines, similar to Airflow. However, it is specifically within the Azure realm and it integrates a lot of different data stores inside and outside of that. ADF powers a lot of the visualization and analysis tools in Azure.

2. What does Azure Data Factory do?

airflow-adf2

  1. It allows you to construct ETL and ELT processes code-free with a nice drag and drop UI. You can open each drag & drop piece and select data sources, or use these pieces as building blocks for operations on datasets - on their way to the destination.
  2. It has more than 90 built-in connectors.

3. Adding Airflow to Azure Data Factory

airflow-adf3

You can orchestrate ADF pipelines with Airflow - and there’s plenty of use cases (see slide above)! It’s orchestrating the orchestrator.

4. Demo: the pipelines in action!

airflow-adf4

During the demo, we demonstrated orchestrating multiple Azure Data Factory (ADF) pipelines using Airflow to perform classic ELT operations, based on the example of various currencies.

In order to see:

  • the entire pipline,
  • the corresponding docker files,
  • the requirements,
  • the code,
  • Airflow version we used, visit a site dedicated to Orchestrating Multiple Azure Data Factory Pipelines in Airflow on our Astronomer registry or Github.

In the Q&A we answered:

  • Can airflow replace the ADF completely?
  • How are you actually checking the status of the ADF pipeline?
  • And other technical questions - watch the video to find out.

See More Resources

Datasets and Data-Aware Scheduling in Airflow

Best practices for running Airflow at scale

How to expertly organize your DAGs with task groups

What's new in Airflow 2.10

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.