Create and use data products with Astro Observe
In Observe, a data product is a composition of assets that deliver a result with business relevance. Observe automatically infers upstream dependencies for the assets that define a data product, like an Airflow dag that populates a dashboard or a Snowflake table powering a recommendation engine.
For guidance on identifying data products in your organization, see how to identify data products in your organization.
The Data Products landing page provides an overview of all your data products and their SLA statuses. You can see at a glance which data products are late or stale, who owns them, and when they were last updated to see a snapshot of data product health.
The following procedures describe how to create data products and leverage them to gain insight across pipelines and deployments. After you create a data product, then you can view data product details.
Prerequisites
Create a data product
Start data product creation
From the Asset Catalog
From the Data Products page
- In the sidebar, go to Observe > Asset Catalog.
- Click + Create Data Product.
Select assets
- In the asset selection panel, use filters to find and select the final downstream assets for your data product.
- As you select assets from the left panel, they will appear in the right preview area.
- Click Continue after selecting one or more assets.
Define data product details
- Enter a Name and optionally a Description for your data product.
- Choose a Data Product Owner: Select either User or Team, then specify the responsible user or team.
- Review your selected assets and inspect the automatically generated asset lineage graph on the right side. You can still add assets to the data product after it is created.
- Click Create Data Product to proceed.
Optionally configure a monitor
See Create a data product monitor for information on monitoring data products.
After completing these steps, your new data product will be available to view and manage under Data Products.
View your data product details
After you create a data product, you can view in-depth details about the performance of your assets by selecting your data product for a closer look.
-
Click Observe in the Astro UI, and then click Data Products.
-
Choose the data product that you want to view details about.
When you see information about a specific data product, you can see summary performance data. This includes general statistics, like the average SLA success rate and information about when the data product was created and last updated. Additional details can be found in the following tabs.
Overview
The overview tab provides summary information about the assets in your data product and the rates of overall SLA success rates, consolidated into daily, weekly, and monthly overall rates. This view allows you to quickly identify trends in historical data for your business-critical pipelines.
Event timeline
Each Data Product in Observe has an Event Timeline that reflects activity from the assets, like dag and task completion, and key issues, like breached SLAs and triggered alerts, in that data product. The event timeline supports filtering by event status along with more fine-grained filters. Selecting any particular event displays metadata about the event, including the notification history of a triggered alert.
The event timeline view allows you to see a record of events associated with your data product. These are categorized into Success, Neutral, and Failure events.
Success events
- SLA Success
- Task Success
- Task Start
Neutral events
- Airflow dataset write
- OpenLineage dataset write
Failure events
- Alert notification
- SLA Breach
- Task failure
For more information about diagnosing failures, see Root cause analysis and AI log summaries.
Assets
The Assets tab shows the assets included in the specific data product.
If you select an asset from the list to examine in-depth, you can see additional asset-specific details. See Assets for descriptions.
Metrics
Data products that include Airflow task assets report the following metrics by default:
- Task retries
- Task failures
- Task duration
Alerts
The Alerts tab consolidates all alerts and SLAs that are specific to your selected data product. From this page you can Add an alert, find an existing alert, and view the notification history for an alert.
