Monitors in Astro Observe

Astro Observe uses monitors to continuously assess the health of your data products and pipelines. When a monitor detects a failure, it can automatically alert your team through your notification channels such as email or Slack. Monitors help you rapidly identify and resolve operational issues in your data workflows.

Data product monitors

The Proactive Failure Monitor monitors a specific data product and sends an alert when any upstream or final dag in the specified data product fails. This enables you to quickly respond to failures, address the root cause, and understand the downstream impact.

When a monitored failure occurs, the alert:

• Provides a direct link to a lineage view, showing the failing DAG and its failing task.
• Identifies all downstream data products impacted by the failure.

Create a data product monitor

Data quality monitors

Data quality monitors help you track the health and integrity of your data assets. You can create monitors for tables and columns to detect anomalies in data volume, completeness, and schema changes. To learn more about data quality and configure a connection, see Data Quality.

Create a data quality monitor

Follow these steps to create a monitor for a table or column.

Row Volume Change

The Row Volume Change monitor tracks changes in table row counts over time to identify unexpected fluctuations in data volume.

When configuring this monitor:

• Specify thresholds based on percentage changes or absolute row-count changes.
• Monitors execute checks according to the schedule you define. If you set the monitor to run every 6 hours, it evaluates whether row counts exceed the configured thresholds within that interval.

Column Null Percentage

The Column Null Percentage monitor tracks the percentage of null values in a specific column to surface completeness problems.

When configuring this monitor:

• Select the column to monitor and define the null percentage threshold.
• The monitor evaluates the column at the interval you specify. If the null percentage exceeds the threshold, the monitor triggers.

Table Schema Change

The Table Schema Change monitor detects when the structure of a table changes, such as when columns are added, removed, or modified. This helps you identify unexpected schema changes that could impact downstream processes.

Custom SQL

The Custom SQL monitor compares the numeric output of a SQL query to a defined threshold. Using this monitor, you can define data quality checks specific to your business requirements (for example, checking for missing records, duplicates, broken joins, or invalid values). Custom SQL monitors can be triggered on a set schedule, manually, or programmatically.

When configuring this monitor:

Common use cases include:

• Flagging duplicate records
• Checking for foreign key violations
• Validating business rules (for example, order amount cannot be negative)
• Ensuring data freshness (for example, last update within 1 hour)

Example SQL queries

1
-- Missing data
2
SELECT COUNT(*) FROM orders WHERE order_id IS NULL
3
4
-- Duplicates
5
SELECT COUNT(*) FROM (
6
  SELECT order_id, COUNT(*) 
7
  FROM orders 
8
  GROUP BY order_id 
9
  HAVING COUNT(*) > 1
10
)
11
12
-- Invalid values
13
SELECT COUNT(*) FROM orders WHERE total_amount < 0
14
15
-- Orphaned rows
16
SELECT COUNT(*) 
17
FROM orders o 
18
LEFT JOIN customers c ON o.customer_id = c.customer_id 
19
WHERE c.customer_id IS NULL
20
21
-- Staleness
22
SELECT DATEDIFF('hour', MAX(updated_at), CURRENT_TIMESTAMP()) 
23
FROM orders

Trigger Custom SQL monitors with events

In addition to a fixed schedule, Custom SQL monitors can be triggered by events. For example, when a specific Airflow dag completes successfully or when data lands in a Snowflake table. This ensures that checks are evaluated immediately after new data lands, allowing downstream consumers to be notified of data quality issues immediately rather than waiting for the next scheduled check.

You can trigger a Custom SQL monitor by calling the following API endpoint:

$
POST https://api.astronomer.io/v1alpha1/organizations/{ORG_ID}/observability/monitors/{MONITOR_ID}/trigger

When triggered, Observe:

1. Runs the configured SQL monitor immediately against the connected data warehouse.
2. Evaluates the result against the monitor’s condition (for example, invalid_rows > 0).
3. Logs the result in the monitor timeline.
4. Sends alerts through configured channels if thresholds are breached.

Example: Airflow DAG trigger

You can include this API call directly in your Airflow dag to trigger validation right after a load completes:

1
trigger_sql_monitor = PythonOperator(
2
   task_id="trigger_sql_monitor",
3
   python_callable=lambda: requests.post(
4
        f"https://api.astronomer.io/v1alpha1/organizations/{ORG_ID}/observability/monitors/{MONITOR_ID}/trigger",
5
        headers={
6
            "Content-Type": "application/json",
7
            "Authorization": f"Bearer {API_TOKEN}",
8
        },
9
        timeout=30,
10
    )
11
)

Chain this operator after your load task:

1
load_data_to_snowflake >> trigger_sql_monitor

to ensure that quality checks run immediately after data load, enabling faster issue detection and downstream notification.

Custom SQL requirements and limitations

• Read-only queries: Only read queries are allowed. Write queries will return an error.
• Query format: Queries must return a single row and a single column. Otherwise, Observe returns an error.
• Numeric output: Query output must be a numeric value.
• Table references: Tables in queries must use the format <database>.<schema>.<table> since the default database and schema might not be set.
• Joins: You can perform multiple join queries as long as they meet the above criteria.
• Rate limits: To prevent system abuse, the following limitations apply:
  • You cannot test the same query more than 3 times in a minute.
  • You cannot test more than 10 queries in a minute.
  • You cannot manually re-trigger the monitor for the next 5 minutes after a trigger request is initiated.
• Initial trigger: Wait at least 1 minute before manually triggering the Custom SQL monitor after the monitor is created.