Are you barely able to understand the DAGs you, or your coworkers, wrote just a few months ago? Do you have issues with DAG parsing times, or mysterious additional database queries? Would you like to implement data-aware pipelines that adapt to your code at runtime but don’t know how? You can avoid these and other common issues by using key Airflow features and following best practices in your DAG code.
Airflow’s Python-based framework makes it easy for even beginners to get started, but if you don’t have a solid foundation, you risk building hard-to-maintain pipelines that are a pain to debug. In this webinar, you’ll learn proven DAG writing best practices including some of the latest Airflow 3 features, that you can use right away to develop faster and reduce errors, including how to:
- Design DAGs that are easier to read, test, and maintain
- Make your pipelines adapt to your data at runtime with dynamic task mapping
- Avoid common pitfalls that can cause performance issues
- Create data-aware pipelines with Assets and event-driven scheduling
Whether you’re writing traditional ELT/ETL pipelines or complex ML workflows, you’ll learn how to make Airflow work best for even your most critical data products.