Using Airflow with Azure Data Factory

WATCH ON DEMAND

Summary:

While Airflow and ADF (Azure Data Factory) have pros and cons, they can be used in tandem for data pipelines across your organization. In this webinar, we’ll cover how using the two together can really get you the best of both worlds!

We'll Cover:

  • The Azure Provider
  • Orchestrating ADF jobs with Airflow
  • How the two tools can be used to help ETL and data quality

Missed the Webinar? Sign up for the Recap

Recap Preview

Welcome to the recap of a webinar on using Airflow with Azure Data Factory!

During the webinar we covered:

  1. What is ADF?
  2. What does ADF do?
  3. Adding Airflow to Azure Data Factory
  4. Demo
  5. Q&A

1. What is Azure Data Factory?

airflow-adf1

Long story short, ADF is an orchestration tool for data pipelines, similar to Airflow. However, it is specifically within the Azure realm and it integrates a lot of different data stores inside and outside of that. ADF powers a lot of the visualization and analysis tools in Azure.

Hosted By

Josh Fell

Josh Fell

Ecosystem Engineer

Josh is an Ecosystem Engineer at Astronomer with a passion for enhancing the developer experience in Airflow. He currently lives in Washington, DC and enjoys playing music and attending concerts in his spare time.

Benji Lampel

Benji Lampel

Enterprise Platform Architect

Benjamin (Benji) Lampel is currently an Enterprise Platform Architect at Astronomer, focusing on how to leverage data quality use-cases with Airflow. He graduated from UC Santa Barbara with a BS and MS in Computer Science, and has worked as a data engineer and principal engineer. In his spare time, he enjoys biking, reading, and going to concerts. Benji currently resides in Brooklyn, NY.