Note: The post Astro Observe is now in Private Preview: Unifying Data Observability and Orchestration was first published in September of 2024, and has been updated to coincide with the public preview release of the product.
As of today (November 5th, 2024) Astro Observe is in Public Preview, building on existing functionality, which reflects hours of thoughtful feedback provided by our design partners. In addition to the prior features released in September, Astro Observe now offers:
-
An Insight-rich UI (updated screenshots below): We’ve overhauled the look-and-feel for Astro Observe, and incorporated dashboard-level visualizations that allow you to quickly understand the performance and health of your data products and their dependencies.
-
Robust alerting: You can now fine-tune alerts across the SLAs you’ve set for your data products and deliver notifications to your team ahead of a breach. Alerts can be configured across supporting tasks and assets within data products, and delivered through the communication channel (Email, Slack, PagerDuty) of your choice.
-
Proactive recommendations: We’re leveraging years of expertise in Airflow best practices to surface optimization opportunities within your data products – including the detection of bottlenecks and anomalous activity across their supportive assets – as well as proactive changes to users’ configurations that can unlock efficiency.
Interested in learning more? Visit the Astro Observe product page for a closer look, including an interactive 30-second demo, additional resources, and to sign up.
Introducing a single pane of glass for your data products
The modern data stack and the workflows that it powers is becoming more complex, and data professionals are forced to grapple with a growing sprawl of dependencies, spanning across multiple teams and deployments. Likewise, as data products play an increasingly critical role in supporting business objectives, most organizations have lost any tolerance for failure or delay in pushing quality data for internal and external consumption.
Today, we’re excited to announce Astro Observe, which brings new and more robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC). Astro Observe is being offered to select design partners as part of a private preview. Users across these platforms can request access to the preview, and receive early access to a centralized dashboard for the management of data products, in addition to a dependency graph that provides a complete view of up-and-downstream dependencies within the data supply chain.
Moving from reactive to proactive interventions
Astro Observe’s Data Products Dashboard gives teams a central platform to assign ownership, manage, and monitor their data products. It creates a clear picture of accountability and enables teams to take action when necessary. Here, teams can set and track SLAs against data freshness and delivery time; and receive alerts when an asset within a data product is at risk of breaching its SLA.
Pictured: Data Product Overview in Dashboard
Pictured: SLA Evaluations view in Data Products Dashboard
Dependency Graphs provide a comprehensive and transparent view of all assets within the data supply chain for a given data product. By viewing the relationships between downstream dependencies and their supported data products, teams can quickly identify and trace issues back to source – reducing downtime and preventing disruptions.
Observe also captures detailed snapshots of dependencies, allowing teams to reference version histories and gain a clear view or job runs or dependencies at given point in time. This perspective not only helps maintain a precise view of lineage for specific dates and times, but also provides valuable insight for debugging, auditing, and managing changes. Users can also develop a historical context of dependencies with the version histories, which surfaces a view of lineage at an exact date and time, providing valuable insight for debugging, auditing, and managing changes.
Pictured: Dependency Graph in Data Products Dashboard
While Astro Observe is agnostic across open source Airflow and other managed services, Astro users enjoy the added benefit of having a single destination for data orchestration and observability. By centralizing all activity to a single interface, teams can understand activity and take action within Airflow without switching between platforms.
Next Up: General Availability
As we move towards GA, Astro Observe users can expect more capabilities, such as a recommendation engine for proactively surfacing actions that teams can take within their for pipelines to improve efficiency and mitigate risk. We’ll also be introducing more functionality around anomaly detection and cost optimization, further allowing users to deliver proactive interventions to improve operations and avoid failures.
In the coming months, Astro Observe will become generally available to all customers (planned for early 2025). If you’re interested in participating in the preview, you can sign up here. We’re excited to invite Airflow users on this journey as we firmly believe that with strong data observability practices comes greater reliability and trust of data products – as well as faster innovation, lower costs, and more productive platform and data engineering teams as critical data assets are better secured and governed.
Additional Resources
