Using Airflow with Azure Data Factory

Watch Video On Demand

Hosted By

  • Josh Fell
  • Benji Lampel

Welcome to the recap of a webinar on using Airflow with Azure Data Factory!

During the webinar we covered:

  1. What is ADF?
  2. What does ADF do?
  3. Adding Airflow to Azure Data Factory
  4. Demo
  5. Q&A

1. What is Azure Data Factory?


Long story short, ADF is an orchestration tool for data pipelines, similar to Airflow. However, it is specifically within the Azure realm and it integrates a lot of different data stores inside and outside of that. ADF powers a lot of the visualization and analysis tools in Azure.

2. What does Azure Data Factory do?


  1. It allows you to construct ETL and ELT processes code-free with a nice drag and drop UI. You can open each drag & drop piece and select data sources, or use these pieces as building blocks for operations on datasets - on their way to the destination.
  2. It has more than 90 built-in connectors.

3. Adding Airflow to Azure Data Factory


You can orchestrate ADF pipelines with Airflow - and there’s plenty of use cases (see slide above)! It’s orchestrating the orchestrator.

4. Demo: the pipelines in action!


During the demo, we demonstrated orchestrating multiple Azure Data Factory (ADF) pipelines using Airflow to perform classic ELT operations, based on the example of various currencies.

In order to see:

In the Q&A we answered:

Ready to Get Started?

Get Started Free

Try Astro free for 14 days and power your next big data project.