Skip to main content

Implementing proactive alerting

Proactive alerting enables teams to stay ahead of data quality issues. You can set up an open-source approach using tools that integrate with Airflow or use Astro, which offers built-in support for data quality monitoring.

Why to use proactive alerting

The use of proactive alerting adds value by helping ensure:

  • Clear expectations for data quality: You can use proactive alerts to establish and evaluate data products against expectations for quality metrics. For example, you can track the accuracy, completeness, consistency, and timeliness of the data in your pipelines.

  • Accountability: By formalizing standards for data quality, proactive alerting establishes accountability. If the data do not meet the agreed-upon standards, the responsible party (whether it’s an internal team or a third-party provider) can be held accountable for remediation.

  • Performance measurement: Proactive alerts provide a framework for setting and evaluating assets against benchmarks and metrics that allow organizations to track performance over time, identify issues, and measure improvements.

  • Proactive management: With proactive alerts in place, organizations can proactively manage data quality. Regular monitoring helps catch and address issues before they impact business processes or decision-making.

  • Risk mitigation: High-quality data are essential for making informed decisions. Proactive alerting helps mitigate risks associated with poor data quality by ensuring that data meet required standards, reducing the likelihood of errors or misleading information.

  • Continuous improvement: Proactive alerting often includes provisions for regular reviews and updates. This encourages continuous improvement in data quality practices and helps teams adapt to changing needs or new challenges.

  • Customer satisfaction: For organizations that provide data as a service, proactive alerting helps build trust with clients by guaranteeing a certain level of data quality. This can lead to higher customer satisfaction and retention over time.

  • Legal and compliance requirements: In regulated industries, maintaining high data quality is often a legal requirement. Proactive alerting helps ensure compliance with these regulations by formalizing data quality standards and monitoring processes.

  • Resource allocation: Proactive alerting helps in resource planning and allocation. By defining expectations for quality, organizations can allocate resources more effectively to meet these standards and more easily handle any issues that arise.

In essence, proactive alerting helps create a structured approach to managing and ensuring data quality, which is critical for effective decision-making and operational efficiency.

When to use proactive alerting

Proactive alerting is most useful when you need to intervene before a table is not delivered on time or an update fails. When performance and governance are not concerns, but you do need to know when a particular task or DAG fails, alerts may suffice. For this use case, Astronomer recommends Astro alerts, which do not require modification of DAG code. Airflow notifications provide some of the same functionality.

By contrast, when you need to implement robust performance and data quality monitoring, a standardized approach at the organizational level, alerting on timeliness and freshness of business-critical data products is recommended. In this case, Astronomer recommend using Astro Observe.

Implementing proactive alerting

Using Astro Observe, you can add proactive alerts to any of your SLAs to get notified when an SLA is at risk of being breached. There are two types of proactive alerts you can set up in Astro Observe.

Data Product Proactive SLA Alerts

These alerts send a notification when a data product asset is approaching its SLA deadline. A common use case is to configure a data product proactive alert on an asset (a task or dataset) that is upstream of the data product. This way you can catch a delay in your pipeline before the deadline of the data product is missed.

For example, you might have a dashboard that needs to be updated by 9am EST every day. You could set up a proactive SLA alert to notify the European team if the 3 most important tables feeding the dashboard have not been updated by 7am EST, giving them 2 hours to investigate and fix the issue before the executive in the US looks at the dashboard.

Data Product Proactive Failure Alerts

These alerts send a notification when a data product asset has failed. Being apprised of an upstream asset failure can help you fix the issue in time to be able to rerun the pipeline for the data product to meet its SLA.

How to set up a proactive alert in Astro Observe

  1. To create a proactive alert, in Astro Observe, navigate to the Alerts tab of your data product. Click + Alert to set a new alert.

    Astro Observe Alerts tab screenshot

  2. Select the alert type from the dropdown menu and select the severity of the alert. Note that to be able to define a Data Product Proactive SLA, you need to have a timeliness SLA set up on the data product.

    Astro Observe Alerts dropdown screenshot

  3. You can define the alert conditions, specifically which assets to base the alert on if they either fail (Data Product Proactive Failure Alert) or are not updated by their SLA deadline (Data Product Proactive SLA Alert).

    Astro Observe Alerts conditions screenshot

  4. Lastly, decide how you want to be notified. If this is your first time setting up a notification on Astro you might need to add a new notification channel by clicking + Notification Channel.

You can now view and modify your new alert in the Alerts tab of your data product.

Astro Observe Alerts list screenshot

Was this page helpful?