Simplified DAG Authoring with New Airflow Features

Summary

In the past six months, many new Airflow features and supporting open-source projects have been developed to make DAG authoring simpler. Utilizing these tools means that you can write less code and design more straightforward DAGs, while still creating efficient pipelines that benefit from Airflow’s best-in-class orchestration.

In this webinar, we’ll cover best practices for implementing important open-source features and frameworks that will simplify the DAG-authoring process, such as:

• How to use datasets – a new Airflow feature that enables data-driven scheduling – to write smaller, more efficient pipelines.
• How to use the Astro SDK – an open-source framework for DAG authoring that allows you to work in SQL or Python while eliminating boilerplate Airflow code – to lower the barrier to entry for new Airflow users.
• When to use dynamic task mapping – an Airflow feature for dynamically generating parallel tasks at runtime – as a simpler alternative to dynamically generating DAGs.