Whether you’re creating complex dashboards or fine-tuning large language models, first, your data needs to be in the right location and the right format. These ETL (extract-transform-load) and ELT (extract-load-transform) pipelines form the foundation of any data product, and Apache Airflow® is the open source standard to orchestrate them.

This eBook covers:

• An overview of decisions to make when designing ETL and ELT pipelines, from methods to pass data between tasks to types of data quality checks
• Key DAG writing best practices including an overview of testing and scaling options in Airflow
• Guidance on how to use Airflow features that can elevate your ETL and ELT pipelines, including dynamic task mapping, data-aware scheduling, and custom task groups