Platform observabilityAstro Observe

Get started with Observe

Astro Observe delivers pipeline-aware data observability purpose-built for Airflow, giving your team visibility into the health, quality, performance, and cost of your business critical data products across the modern data stack. This guide outlines the fundamental requirements and steps to onboard.

Prerequisites

  • Astro Deployment running Astro Runtime 9 or later (apache-airflow>=2.7.0)
  • OpenLineage Airflow provider (apache-airflow-providers-openlineage>=1.12.1) and OpenLineage client (openlineage-python>=1.38.0) specified in project dependencies (for example, in requirements.txt)
  • At least one asset running in Airflow
  • Observe Member, Billing Admin, or higher role required to create data products (see user permissions reference)
  • OpenLineage enabled for RE agents if using Remote Execution

Astronomer recommends using the latest possible OpenLineage provider version and the latest OpenLineage client version. The client can be updated regardless of your Airflow version. See Configure OpenLineage on Astro for more information and upgrade instructions.

Set up Astro Observe

1

Enable and verify OpenLineage versions

Confirm your Deployments are running the recommended OpenLineage client and provider versions by checking your Astro project’s requirements.txt. See Configure OpenLineage on Astro for instructions on upgrading to the recommended client and provider versions.

2

Review assets and lineage

  • Check that the Asset Catalog includes assets you expect. Observe captures Airflow assets based on run data from the last 90 days.
  • If you are missing assets for dags and tasks that ran in the last 90 days, ensure OpenLineage is enabled in your Deployments. See how to enable or disable OpenLineage.
  • If OpenLineage is enabled, check whether the assets are produced by jobs with supported operators. Supported operators and hooks are listed in the OpenLineage documentation.
  • For operators not supported out of the box, or for custom operators, see options for emitting OpenLineage events.
3

Create your first data product

A data product is a composition of assets that, taken together, deliver a result with business relevance.

For guidance on identifying data products in your organization, see how to identify data products in your organization.

In the Astro UI, navigate to Observe > Data Products. Select + Data Product and follow the prompts to define your product and choose assets.

See Create a data product for more information.

4

Set a Service Level Agreement (SLA)

Add an SLA to your data product to monitor freshness or timeliness of your data product. See Create an SLA.

5

Configure alerts

Set up alerts to be notified of SLA violations or potential failures. To monitor the SLA you just created, click the SLAs tab in your data product, and select the SLA you want to monitor. Click + Alert to set up a Data Product SLA Violation Alert. See Create an alert for details on alert configuration settings.

6

Grant permissions to team members

To allow others to create data products, SLAs, and monitors in Observe, grant the Observe Member or Observe Admin role to any teammates who need to set up and manage these features. See user permissions reference for instructions on managing roles.

Next steps and resources