Scale Airflow with confidence using Astro’s new Alerting capabilities

Alerting is a powerful new tool in Astro designed to help you proactively monitor and manage your DAGs. This new addition to our suite of data observability solutions is built on top of OpenLineage, allowing you to define alerts based on DAG run failure and task duration.

Without the ability to proactively monitor and manage business-critical DAGs, you’re forced to react to issues after they have occurred, resulting in costly downtime and lost productivity– which can in turn result in lost revenue and unhappy customers.

With Astro’s new alerting feature, you can now define alerts on DAG run failure and task duration. These alerts can be easily integrated with popular tools like Slack and PagerDuty, ensuring that you are notified in real-time when an issue arises through the channels you are already monitoring.

Imagine you have a DAG that processes data for a critical application (if you’re a data engineer, you probably don’t need to use much imagination here). You want to be alerted immediately if there’s an issue with the DAG, or if the tasks within the DAG are taking significantly longer than expected. You can configure alerts on Astro to notify you immediately via your channel of choice so that you can take action, if necessary. These alerts help ensure you don’t get that 9am call about some key data missing!

We’re launching our Alerting capabilities with two alert types to start:

• DAG Failure alerts: receive an alert as soon as a DAG fails
• Task Duration alerts: receive an alert when a task takes longer than a defined duration, as soon as the SLA is breached - not when the task is completed

While open-source Airflow has support for DAG failure and task duration alerts, they function a bit differently. Defining your alerts in Astro allows you to centralize your alert channels and rules across your deployments, instead of just within each DAG. Additionally, task duration alerts get triggered immediately after your specified duration, not at the end of the task like in open-source Airflow. Astro’s alerting functionality will make it easier to manage your alerts at scale and get notified about issues immediately.

To access this new feature, simply log in to the Astro UI and navigate to the Alerting tab under Workspace Settings. Clicking “Add Alert” on the upper right will take you to the New Alert creation page:

In this case, I want to create a Pipeline Failure alert. Now, I just need to select the DAG (or DAGs) I care about to the alert.

This is just the beginning of what’s possible with DAG alerting. In the future, we plan to leverage OpenLineage data to augment alerting capabilities at the dataset level, and provide auto-resolution capabilities.

We’re thrilled to offer this new feature to Astro users and are excited to see the impact it will have on data observability workflows. If you’re interested in learning more about DAG alerting, check out the docs. And, if you want to try it out for yourself, you can sign up for a free trial of Astro today!