27,000 daily tasks driving 18+ data products. All overseen by just 5 team members. Sound familiar? As data pipelines grow in scale and complexity, maintaining reliability and ensuring timely delivery becomes increasingly difficult.
In this webinar, Maggie Stark, Staff Data Engineer at Astronomer, will share how Astronomer’s data engineering team reduced DAG failure rates by 81%. You’ll learn actionable tactics for increasing pipeline reliability including how to tackle upstream and cross-DAG dependencies and how to proactively identify anomalies before failures occur.
Get the strategies the team used, including how to:
- Switch to Airflow Asset scheduling to resolve upstream data dependency issues
- Orchestrate cross-DAG dependencies and failure triage by implementing a Control DAG
- Implement centralized data observability to monitor SLAs, prioritize incidents, and troubleshoot faster with real-time lineage and log insights
Whether you’re scaling Airflow in production or just starting to think about observability, you’ll walk away with actionable tips to increase pipeline reliability and reduce engineering firefighting.
