The latest insights from our team of Apache Airflow® experts.
Otto by Astronomer uses AI to convert Control-M, AutoSys, and Automic (UC4) job definitions into production-ready Airflow Dags. Faster migrations, less technical debt.
Bloomberg's Data Platform Engineering team shares four real-world Airflow configuration fixes that resolved production issues across 2,500+ DAGs — no code changes required.
Astronomer’s Customer Reliability Engineering team was recently honored to receive an award by Business Intelligence Group for our support, a step toward our audacious goal of being the best software support team in the world. We are proud to work with such an incredible customer base, many of whom use Astro for mission-critical workloads. That responsibility demands an uncommon level of service and excellence.
Astro Private Cloud 2.0 in Generally Available
Uber is migrating 200k pipelines to Airflow 3. Here's the architecture, tooling, and strategy behind one of the largest orchestration migrations ever undertaken.
Otto is the only data engineering agent purpose-built for Airflow, and the first to bring Astronomer''s depth of operational Airflow knowledge directly to the engineers doing the work.
apache-airflow-providers-common-ai 0.1.0 shipped in April 2026, Airflow's official provider for AI and LLM workflows, built on PydanticAI and sharing its foundations with the airflow-ai-sdk. Same decorators and engine, plus toolsets that turn 350+ provider hooks into agent tools, durable execution, and built-in human review. Here's what's new, why it matters, and how to migrate without losing a weekend.
Disaster Recovery on Astro is Generally Available. A look at the architecture decisions, trade-offs, and engineering lessons behind cross-region Disaster Recovery on Astro - from database replication with Aurora Global Clusters to warm standby compute and one-click failover
The richest source of context for AI agents already exists in your stack. It's your orchestration layer. Here's why static metadata alone won't get you there.
Cosmos 1.14 delivers significant improvements to Watcher execution mode, the fastest way to run dbt in Airflow, plus a fully restructured documentation experience built around how teams actually use Cosmos in production.
See how Kaiser Permanente runs AI and ML research pipelines on-prem with Airflow, and how Astronomer supports regulated industries with the same flexibility
See how SAP built a production RAG pipeline with Apache Airflow to power Joule for Consultants, processing 5M+ documents across 15 data sources for the enterprise.
With Blueprint, creating Airflow pipelines is now possible for anyone in your organization. Data engineers define templates, and others can create pipelines though a drag-and-drop no-code interface in Astro. No python or Airflow knowledge required.
Learn how Blueprint and the Astro IDE let platform teams define governed templates so anyone can build Airflow pipelines without writing Python.
Asset partitions, async tasks, and continued improvements to the Airflow 3 platform
How the Astronomer Education team uses AI to help respond to user tickets and answer questions about Academy usage.
See how Pittsburgh's two-person data team uses Apache Airflow and Astro to power open government data, cut manual work, and deliver real civic impact.
How Astronomer built an AI agent that reads every Apache Airflow commit daily to catch breaking changes before they reach customers.
DoorDash runs one of the largest Airflow deployments in the world. See how they built the Orchestration Frederator to scale without the chaos.
How Red Hat built a trusted data and AI platform on Apache Airflow and Astro, powering 95 data products and production AI agents across the enterprise.
The Astro CLI is the fastest way to develop and deploy Airflow pipelines from your own editor. Today we're making it more powerful with standalone mode, automatic port management, and direct API access.
Amex runs 100+ production Dags orchestrating metadata with Airflow. See how they built the operators, pipelines, and governance framework that makes it work.
Today, we are announcing the public preview of cross-region Disaster Recovery on Astro, now available on AWS.
How Astronomer's new dataplane-based auth architecture improves Astro resilience, reduces blast radius, and keeps DAG runs running during control plane outages.
Master Apache Airflow's six SQL check operators to build comprehensive data quality checks. Learn when to use each operator and how to sequence them effectively.
Visa cut BI dashboard refresh times from 24 hours to 2 hours with Airflow. Learn how they automated end-to-end workflows and achieved zero missed SLAs.
How Astronomer built Kepler, an internal Slackbot and CLI tool that answers ad hoc data questions using LLMs, hybrid search, and code context.
How a Principal Product Manager used AI coding agents like Claude Code during onboarding to automate workflows, track customer conversations, and ship features in his first 90 days at Astronomer.
Datadog processes 100+ trillion events daily. Learn why they chose Airflow 3 for multi-tenancy, and data-aware scheduling at scale.
Dag-level roles are now available on Astro, giving Enterprise customers the ability to control access to individual Dags within a shared deployment.
Best practices from AI at Oracle: use Airflow to scale GPU-heavy AI and MLOps on Kubernetes with higher reliability, visibility, and GPU ROI.
How Expedia runs 14,000+ Dags across 200+ isolated Airflow clusters with multi-tenancy, templates, and automated CI/CD.
Whether you use Claude Code, Cursor, VS Code, or any of the 25+ compatible AI coding tools, you can now access specialized Airflow knowledge right where you already work.
Learn how to build a decision-tracing context graph with Apache Airflow using the HITLOperator, Airflow AI SDK, and a Slack plugin to capture human decisions.
This year’s winners span industries, regions, and use cases, but they share a common thread: data orchestration as a strategic advantage.
How Qualcomm uses Airflow 3 to orchestrate chip design at massive scale, running millions of EDA tasks daily across global data centers.
Whether you're building new automation or migrating from the beta API, this is the stable foundation you need to programmatically manage Astro at scale.
How the Financial Times uses Apache Airflow to orchestrate AI pipelines, structure messy public data, and power investigative journalism at scale.
Data Quality monitoring in Astro Observe is now available in public preview with powerful new capabilities that fundamentally change when and how you validate data.
We've just released the State of Airflow 2026 report, built on insights from over 5,800 data professionals across 122 countries—the largest survey of data engineers ever conducted.
This new cohort brings together an outstanding group of data engineers, architects, and platform leaders from around the world who share a deep commitment to Apache Airflow® and to advancing the practice of data orchestration.
Scale your data engineering team with a declarative Airflow framework and the write-audit-publish pattern, making pipeline development fast, safe, and reliable.
There’s an urgent reason why localized orchestrators are a strategic mistake. One that has nothing to do with exit costs or vendor leverage and everything to do with what AI needs to work.
Discover how Duolingo uses Airflow to orchestrate AI content pipelines, automate critical workflows, cut costs, and scale learning to millions.
Learn how Deutsche Bank runs Apache Airflow in a highly regulated environment to orchestrate audited, hybrid banking workflows at scale.
You’ll learn: The 3 big reasons for upgrading to Airflow 3, how to plan your Airflow upgrade, and more.
Astro CLI and Astro IDE serve different feedback loops. Learn when to use each—and how combining them helps teams build and ship Airflow faster.
By treating your critical pipelines as products, you make the invisible visible, and you might just get the budget (and the sleep) you deserve.
We’ve made Astro IDE even more powerful, with a set of improvements providing the agent with more context, and data engineers more control over how the agent responds.
A guide to understanding data observability, why it matters, and how modern teams are using it to deliver trusted, reliable data at scale.
How GitHub uses Airflow for open source insights, customer success and continuous Copilot improvements plus nine years of lessons from its data platform.
How OpenAI scaled Apache Airflow into a unified orchestration platform, boosted reliability, and replaced legacy schedulers across the business.
Learn how to effectively use AI tools like the Astro IDE to write maintainable Airflow DAGs by providing proper context, using structured prompting techniques, and enforcing governance standards rather than relying on "vibe coding."
Build multi-agent AI pipelines with Airflow, orchestrating specialized agents, RAG with Aryn AI and Weaviate, human-in-the-loop steps, and Kafka.
This release focuses on helping teams run dbt workflows faster while maintaining the task-level observability that makes using Cosmos superior to running dbt projects with the BashOperator.
In this second part, we explore the middle ground between full code and full abstraction: configuration-based authoring with DAG Factory.
Teams can move from reactive alerts to proactive prevention, detecting issues at the source before they break dashboards or delay critical SLAs.
The Astro IDE allows you to run your first Airflow Dag within X minutes! Can you beat my time?
Model the true cost of Airflow. Use Astronomer’s TCO Calculator to uncover savings and ROI from a fully managed, unified orchestration platform.
We look at how abstraction reshapes orchestration, why it matters for technical managers, and how to position teams to thrive in a multi-level reality.
Astro Private Cloud delivers enterprise-grade data orchestration with control plane and data plane separation, enhanced security, and Apache Airflow 3 support.
As part of the Astronomer DevRel team I get to explore new features in the weeks before the release to produce educational content, for humans but also for our friendly helpful robots: AI coding assistants.
The momentum continues from the release of Airflow 3
Since launching in private preview earlier this year and over the course of the public preview, design partners across industries – from automotive to travel, logistics, education, and sports – have been putting Astro IDE to work.
Introducing the Astro Executor, the Airflow 3 architecture built for performance and reliability. Get 70% higher concurrency, fewer failures, and lower infrastructure costs.
Here we dive into why + how the Data Team built scalable, maintainable data quality checks inside the developer experience and made maintaining high data quality an attainable outcome.
Challenges of multi-tool observability, Airflow's limited features, and how Astro Observe offers comprehensive pipeline monitoring in one platform.
Today we're announcing Astro IDE, the first AI-powered IDE purpose-built for Apache Airflow.
DAG Factory, the open-source tool for declarative DAG authoring in Apache Airflow, reaches a major milestone with version 1.0.
Learn the truth about time-based scheduling limitations, ML/AI workflow compatibility, streaming processing, and whether Airflow is outdated technology.
In this second post, we’ll turn our attention to another common theme in discussions about Airflow: its architecture and performance.
Our team has been focused on pipeline reliability and operational confidence. We'll share our journey: where we started, where we struggled, and where we are now.
Debunking some common Apache Airflow misconceptions.
We’re Astronomer’s in-house data team. We center around two main goals: make data valuable and reliable at Astronomer , all with Airflow!
Get your free ticket to explore how data engineering is evolving from internal analytics to mission-critical applications!
Cosmos 1.11.0a1 introduces alpha support for dbt Fusion—the next-generation dbt engine that unlocks lightning-fast parsing, state-aware orchestration, and real-time validation, all orchestrated natively in Airflow.
While LM Arena leaderboards grab headlines, an infrastructure battle is happening in the orchestration systems that transform AI from lab experiments into production solutions that solve real-world problems.
Discover how Procter & Gamble uses Apache Airflow to modernize data systems, power AI, and streamline legacy workflows at global scale.
Astronomer is excited to announce enhanced data quality monitoring in Astro Observe, now available in private preview.
Meet Cohort 5 of the Astronomer Champions Program for Apache Airflow in 2025. Data leaders from top global organizations are driving data engineering's future.
With Remote Execution on Astro, enterprises no longer have to choose; you can now run workloads exactly wherever they need to be, while still benefiting from centralized orchestration and observability in Astro.
Discover why context is vital for effective AI implementation and how shifting from centralized data systems to context-rich approaches enhances decision-making.
Reimagining data orchestration for the AI-era
Cosmos now natively supports source node rendering! Visualize your dbt DAG more effectively with this enhanced feature. Learn how it simplifies data lineage.
In the AI era, orchestration platforms become the lynchpin.
The Digital Operational Resilience Act (DORA) addresses the critical need for the financial sector to effectively manage digital operational resilience in the face of rising cyber threats and Information and Communication Technology (ICT) disruptions.
Introducing the AI SDK for Apache Airflow™
Astro Enhanced Alerting provides a unified monitoring experience through pattern-based alert rules that teams can apply seamlessly across hundreds of DAGs simultaneously.
Astronomer and IBM join forces to revolutionize enterprise data orchestration! Discover how this exciting collaboration will transform data management.
Vibrant Planet optimizes Airflow pipelines for wildfire prevention, using ML and dynamic memory scaling to process massive geospatial datasets
You can now upgrade to our Team plan while continuing to only pay for what you use, billed monthly to a credit card or cloud marketplace.
Airflow 3 is the biggest release in Airflow’s history, the result of a massive effort from the global Airflow community.
Key Findings from the State of Airflow Report 2025
Integrating Slurm and Airflow, Meteosim runs over 6,000 pipelines daily with zero failures. Read the post to learn how they did it.
How Laurel uses Airflow to scale GenAI, automate timekeeping, and cut LLM costs by $500K per year. Learn how DAGs power model retraining and inference.
The new standard for pipeline reliability and data product observability.
FreightWaves transitioned from a scattered and inefficient setup to Astronomer’s Astro, powered by Apache Airflow, to transform their data engineering practices.
Discover Rakuten Kobo’s strategies for scaling Airflow usage across teams, fostering collaboration with guardrails, alerts, and shared environments.
DataOps is the future of the modern data stack. Learn how orchestration-first platforms like Astro are unifying the stack & delivering value.
Discover the winners of the Astronomer Data Excellence Awards! Celebrating groundbreaking achievements in data orchestration with Astro and Airflow.
BAM simplifies complex data pipelines with dbt on Airflow using Cosmos, enabling teams to scale transformations with ease and transparency.
How to Migrate Legacy ETL Workloads from Informatica to Airflow DAGs Using DAG Factory
Discover Autodesk’s secure UAT environment for PII testing, achieving faster deployments and reduced technical debt, powered by Airflow and Astronomer.
TrackFly turned to Astronomer’s managed Airflow solution, Astro, to streamline their ETL (Extract, Transform, Load) workflows and focus on core business goals.
Wix runs 5,500 Airflow pipelines daily, powering ML workflows for the company that hosts 10% of the web. Explore their innovative MLOps platform.
Learn how Airflow powers Cloudflare’s infrastructure automation, from server diagnostics to GPU inference clusters in 100+ data centers
Learn how Circle manages stablecoins across multiple blockchains with Airflow, tackling MWAA challenges and exploring Astronomer for speed & scale
Learn how Robinhood scaled mission-critical financial workflows with Airflow, from trading to clearing, migration strategies, and future plans.
Discover how Instacart scaled Apache Airflow to handle 2,200 pipelines and 16 million tasks monthly with custom tools, IaC, and centralized management.
Explore how the Texas Rangers and Philadelphia Phillies use Airflow to revolutionize data orchestration and ML, driving faster insights and a competitive edge.
Explore core data observability features: analytics, monitoring, and alerts. Gain insights into pipeline health, prevent failures, and ensure data reliability.
Learn how Panasonic leverages Apache Airflow to streamline battery production, scale workflows, and plan for streaming and Kubernetes.
Learn how to build an ELT pipeline extracting data from S3, loading it into Databricks, and transforming it with notebooks using Apache Airflow.
Discover how ASAPP turbocharges MLOps with Airflow and Spark, scaling LLM workflows to boost efficiency & achieve lightning-fast processing in hours vs days.
Discover how LinkedIn uses Airflow to orchestrate 12k pipelines managing 1m deploys for 7k services. And Airflow 3 will make provisioning even easier!
Learn how Stripe processes petabytes of data daily with Apache Airflow, ensuring compliance while accelerating developer workflows with its dev/test tooling
Go on an Airflow journey with Bosch as they scale from 1k to 50k DAG runs per hour and 1.2 million pipeline runs per day with an average 1 second latency.
Autodesk scales with Astro! 🚀 Learn how they migrated from Oozie to Airflow, boosting efficiency, scalability, and data-driven decision making.
Here are four key observability insights to consider focusing on when setting up an observability solution: data freshness, on-time delivery, data dependencies tracking, and data quality.
Learn why Bloomberg selected Airflow over Dagster and Prefect, reducing the ETL of 50 million loans and 5 billion data points by 50%
Discover how Burns & McDonnell scaled data delivery from ad-hoc workflows to a unified platform using Apache Airflow, providing reliable data in under 24 hours.
Today, we’re excited to announce Astro Observe, which brings new and more robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC).
Prevent Airflow downtime with proactive deployment health alerts. Automate monitoring & get actionable insights for reliable data pipelines.
Discover how Uber streamlined data workflows with Apache Airflow, scaling to support 1,000 teams, 450,000 daily pipeline runs, and plans for Airflow 3
Anyscale and Astronomer join forces to solve the challenges of scaling machine learning and AI.
This final cohort for the 2024 calendar year brings together data leaders from Fortune 500 companies around the globe, all possessing extensive Airflow expertise.
Astronomer launches Vulnerability Disclosure Program, a way for us to better engage with the global community in receiving, recognizing, and rewarding findings from the collective security community.
How Isolated Environments Enhance Execution, Security, and Productivit
We're happy to announce that the newest version of Astronomer’s dbt-core integration, Cosmos 1.6.0, is now available, featuring a list of enhancements and some great new additions to serve the community.
Announcing the Astro Terraform Provider! Use Terraform to manage and automate your Astro deployments.
Explore key findings from the 2024 Gartner® DataOps Market Guide. Enhance your data operations with these insights.
This tutorial provides step-by-step instructions on how to set up a data ingestion pipeline to automatically ingest data from S3 into Snowflake, running in production.
Join Astronomer at Airflow Summit 2024! Explore Astro, attend expert sessions, and network at our exclusive afterparty. See you there!
A Cost Sensitive Approach to Scalable Model Personalization Through Airflow
Explore how data products are evolving, the challenges of modern data pipelines, and the future of unified orchestration and observability.
Recent skepticism about Generative AI is healthy, for the short-term impacts are often exaggerated. But the long term impacts are most certainly profound.
Like a conductor, Airflow understands the flow of data, how a network of operations comes together to yield a data product.
The way we develop, orchestrate and observe data products needs to change. Learn how to get started by downloading our new guide.
The Airflow 2.10 release brings greater flexibility and expansion of some of the most widely used Airflow features.
Learn to automate Astro onboarding with Terraform! This tutorial shows you how to create and manage workspaces.
Introducing a unified approach to orchestrating dbt and Airflow with Astro
We are excited to announce the new release of the Astro Platform, introducing exciting new features designed to enhance your data orchestration experience.
Discover the intricacies of Airflow trigger rules with visual examples and practical applications. Learn how to define and use various trigger rules to optimize your DAGs efficiently in Airflow. Essential reading for Airflow users working with version 2.9.2.
We are thrilled to announce that Astronomer is officially taking over Adam Boscarino’s DAG Factory, an open source project that allows DAGs to be generated from YAML files.
We are thrilled to introduce Cohort 3 of the Astronomer Champions Program for Apache Airflow!
Explore Airflow 2.9's key features in Astronomer's 29 days of Airflow 2.9. Learn about dynamic scheduling, task management, and more.
One of the biggest questions we get asked when discussing data orchestration for GenAI is how to get started. That is what our new GenAI Cookbook is designed to answer.
Announcing The Data Flowcast: A podcast dedicated to all things Apache Airflow. Tune in for expert insights and trends.
SnowPatrol helps Grindr save $600k in Snowflake costs. Discover how to use ML for anomaly detection and cost management.
Learn how Apache Airflow® and Astronomer streamline data and model orchestration for Generative AI success. Explore practical use cases and a comprehensive guide.
The Airflow 2.9 release brings significant enhancements to user-favorite features like data-aware scheduling, dynamic task mapping, and object storage.
An introduction to data pipeline testing strategies, best practices, and implementation techniques.
Our beta cohort of 10 is now joined by 23 hand-selected individuals who, we believe, truly embody what it means to champion the Apache Airflow® Project.
Welcome to the latest Astro Platform release — we’re thrilled to introduce enhancements aimed at bolstering governance at scale and across environments, fortifying the security of your data platform, and accelerating innovation.
Learn how SnowPatrol leverages machine learning and Airflow to detect anomalies in Snowflake usage, optimize costs, and improve data pipeline efficiency.
Our dedication to delivering top-tier products drove us to integrate Hybrid Search from Weaviate and the Cohere Rerank into our existing Ask Astro system.
One year ago, we rolled out a new incident management process. Read on to find the past, present, and future of incident management at Astronomer.
Discover how Dosu leverages Astronomer to streamline data orchestration for AI applications, ensuring reliable pipelines and boosting productivity. Learn how this partnership enhances AI development and supports open-source communities.
Discover how we created an efficient release note system with Towncrier. Minimized friction and improved developer workflows.
ADF excels in creating quick and low-code data jobs with an intuitive UX. By layering in Airflow’s expressiveness orchestration capabilities on top of ADF workflows, developers get end-to-end visibility of their workflows without needing to migrate any jobs.
Astronomer has moved! At the start of this year, we relocated our headquarters to the heart of New York City at 50 West 23rd Street to support our growing business and customer base
A walkthrough of Astro's architecture and deployment model
The value of Astro: a summary of Forrester’s Total Economic Impact™ Study
A demonstration of how a platform team can develop a template Astro project for bootstrapping Astro projects for development teams. We demonstrate how to use Cookiecutter for developing a template project and Cruft for synchronizing generated projects with changes in the template project.
In this post we will cover how we had implemented a solid MLOps pipeline and data orchestration with the help of Airflow in multiple use cases.
Today, we're thrilled to announce the launch of the Astronomer Champions Program for Apache Airflow®, a global initiative designed to recognize and empower outstanding data practitioners who are dedicated advocates of this powerful open-source orchestration tool.
The latest minor Airflow release includes new features and improvements such as the Airflow ObjectStore, Listener hook for Datasets, enhanced logging capabilities, and more.
The new Deploy Rollbacks feature enables users to revert any code deployed to Astro Deployments, including upgrades, to a known "good" state. This allows users to quickly recover from failing pipelines and avoid critical downtime.
Master Airflow's Trigger Rules & XComs for flexible, resilient data pipelines. Learn how to handle complex scenarios and ensure flawless workflow execution.
Unveiling Astro's latest features for streamlined connectivity, confident upgrades, and cost-efficient scaling. In this article, we’ll dive into these key features and explore how they can benefit your organization.
Experience advanced authentication with Apache Airflow®™ on Astro, the Azure Native ISV Service. Securely orchestrate data pipelines using Entra ID. Follow our step-by-step guides and leverage open-source contributions for a seamless deployment experience.
Build production-ready ML applications with Airflow's integrations for LLMs and AI.
Explore the TaskFlow API and traditional operators and find out how to combine them for dynamic, efficient DAGs.
See how the new Tecton Airflow Provider can make your feature pipeline orchestration within Apache Airflow® more efficient.
Data ingestion for RAG with LLMs: Ask Astro Part 3 covers vector stores, schema design, and chunking.
The state of deploying pipelines with dbt has changed considerably in the last few months. Over the last few weeks, I was working with Astronomer to test out their new tool, Cosmos, to deploy dbt workflows onto Snowflake.
Databricks vs Airflow from a production management perspective. Explore the differences in setup, monitoring, integrations, scalability & customization.
An example project showing how to use Apache Airflow® to orchestrate a machine learning pipeline with the Snowpark provider and Snowpark ML.
Learn how easy it is to migrate Python scripts to Airflow DAGs, streamline orchestration, and leverage Airflow features to boost job efficiency.
Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.
Build an LLM-powered chatbot with Airflow! Learn how to leverage domain-specific knowledge to create intelligent applications like "Ask Astro." Astro LLM meets Apache Airflow.
Explore how Apache Airflow® solves day-2 operations challenges for LLM applications. Learn about scalable, reliable, and auditable workflows with Airflow.
Dive deeper into Airflow CDC implementation. Explore advanced use cases, best practices, and handle schema evolution & log-based sync effectively.
Choosing the right data orchestration tool for your needs can be tough. This blog post compares Databricks Workflows and Apache Airflow, two popular options.
Learn how to debug Airflow DAGs in 3 key steps. Eliminate common issues, set up a local development environment, and implement testing for seamless Airflow workflows.
In the realm of machine learning, managing workflows efficiently is paramount. One tool that has emerged as a game-changer in this space is Apache Airflow®.
Understand the basics of Change Data Capture (CDC) in Airflow. Learn its importance, use cases, and core concepts for data pipeline success.
The latest minor release includes several new features, such as automatic setup/teardown of tasks, built-in OpenLineage support, cluster activity view, fail-stop functionality, and more.
Get insights on how to use Apache Airflow® and the Kubernetes Executor for data processing, along with proven best practices and tips for scaling your workloads.
The Local Upgrade Test command in the Astro CLI eliminates upgrade pains and ensures safe upgrades, allowing users to confidently identify and resolve compatibility issues, and DAG import errors.
In this blog post, we will dive into the details of the Astro’s Role-Based Access Control (RBAC) and new Workspace Role updates, and explore improvements to popular use cases of Astro.
Optimize your data pipelines with Apache Airflow®. This guide covers tips for faster, more reliable, and easier-to-manage ETL workflows.
Use Apache Airflow® with CrateDB to run ETL processes, deploy with ease thanks to Astro and CrateDB Cloud.
The easiest way to orchestrate dbt Core using Apache Airflow®
The Astronomer Approach to Clear and Effective Technical Documentation
Learn how to effectively monitor Airflow DAGs, track SLAs, and maintain data pipeline health. Explore Airflow UI, notifications, and advanced observability tools.
Maximize ETL efficiency with hosted Apache Airflow® on Astro, not self-hosting open-source. Benefit from simplified infrastructure management, scalable elasticity, and dedicated support for your workloads.
This guide covers best practices for everything from choosing the right Airflow deployment model to configuring your DAGs for optimal performance.
The new provider will make it easier for organizations to use Airflow to automate and manage their Fivetran pipelines.
Announcing the Astronomer and Snowflake partnership! Transform your data pipelines with Snowpark and Airflow.
Use Apache Airflow® with DuckDB and MotherDuck in three different ways. Access the DuckDB Python package directly, leverage the DuckDB Airflow provider, and use DuckDB with the Astro Python SDK.
Use ~1,000 open-source Airflow operators and define your own custom operators in the Astro Cloud IDE with the newly released cell type functionality
Beyond MWAA: Top 7 data orchestration tools for optimized workflows and increased productivity.
Picking the right tools for your data stack depends on your exact business and engineering needs, and the choice may seem daunting. Thankfully, there are several popular tools, each with thousands of users, all with a unique approach for managing data pipelines.
Migrating the Astronomer Registry’s backend from Airtable to Postgres and a Golang REST API
This new functionality in Astro lets you easily implement DAG-level or task-level alerts to be sent to Slack or PagerDuty.
Apache Airflow® 2.6 contains over 500 commits from over 130 contributors, adding up to 35 new features, 50 general improvements, and 27 bug fixes.
Announcing Kubernetes Executor support in Astro. You can now take advantage of the power of Kubernetes to manage resources & scale your Airflow workloads.
Ensure that all code changes are deployed within your CI/CD processes, increase code quality, and enforce automated testing.
Learn why OpenLineage is catching on and see what lies ahead for this open-source standard.
Authorized Workspaces, a new feature in Astro, lets customers isolate teams or projects to specific clusters in their data planes.
The Kubernetes Executor offers Astro customers task isolation, efficient resourcing, and simplicity.
Simplify data quality checks in Airflow with Great Expectations. Learn how to integrate, set up, and leverage its powerful features for reliable pipelines.
The updated Astro homepage brings together the key pieces of information a user needs to start their day.
Find out how Airflow has been optimized in 2022. Learn about major updates, including data-driven scheduling, dynamic task mapping, and UI enhancements.
Learn how to win a full scholarship to CoRise’s new Airflow and data orchestration course.
The new DAG-only deploy feature in the Astro CLI makes deploys to Astro significantly faster and allows for more flexibility in CI/CD workflows.
Learn how improved DAG-testing commands in the Astro and Airflow CLIs make DAG authoring easier and help DAGs run more reliably.
Find out what the most popular and useful DAG views in the Airflow UI are. Learn about the Airflow Graph View, Grid View, Calendar View, and Browse Tab.
Discover the Astro Cloud IDE, a notebook-inspired tool for writing data pipelines. See how to define tasks and connections without knowing Apache Airflow®.
Check out what’s new in Apache Airflow® 2.5. Learn more about improvements to Airflow’s dynamic task mapping and data-dependent scheduling features.
Learn about Airflow & its updated features. Get to know how users can benefit from Taskflow API, Custom XCom Backends, Astro SDK, and the Astro Cloud IDE.
Learn how to securely hook up data sources and implement strong authentication in Astro — a modern data orchestration solution powered by Apache Airflow®.
Learn how to extract data lineage events from your Airflow pipelines using OpenLineage. Plus, see how these three methods work with the Astro platform.
Discover how Astro can help you understand, communicate, and solve pipeline problems. Learn about the key pipeline observability feature – the Data Graph.
Hear from Julien Le Dem, Chief Architect at Astronomer, about the creation of OpenLineage and how it’s evolving into a standard for data lineage.
Learn more about Airflow-driven data quality checks, their benefits, and design. Find out how data quality issues are detected and solved at Astronomer.
Learn what the new upgraded Astro Python SDK 1.1 offers to Airflow users. Find out more about data-driven scheduling, dynamic tasks, and Redshift support.
Learn how Astro, the modern Airflow-powered data orchestration platform, helped Astronomer build a fully coordinated data ecosystem.
Learn how to tune the data system and have critical data products ready on time with micropipelines. See how to make DAG authoring easier with Airflow 2.4.
Hear from Steven Hillion and Taylor Merrick about how our data scientists combine tooling and process to encourage company-wide data product development.
Learn how Astronomer is using the new data-driven scheduling feature in Airflow 2.4. See how it benefits DAG authors and helps solve timing problems.
Learn all about the Astro CLI — the free, open source tool that makes it easy to install, run, and test Apache Airflow® from your command line.
Discover the newly released Apache Airflow® 2.4. Find out how its new data-driven scheduling logic enables faster and easier delivery of data.
Hear from Astronomer’s Senior Vice President of R&D about how data orchestration and observability improve the quality and reliability of dataflows.
Two more reasons that users who need high levels of data security and protection can count on Astro.
Find out how we use data to keep track of what’s happening in the Airflow project.
Discover the Astro Python SDK—an open-source framework for writing Airflow data pipelines.
Astro is now available on AWS, Microsoft Azure, and Google Cloud in 47 regions across six continents. Learn more on our blog.
Learn about Airflow 2.3’s new grid view. Find out how to easily visualize complex representations in Airflow’s UI with this long-awaited intuitive feature.
Learn about Astronomer Providers — a collection of open-source operators, hooks, and sensors that allow you to schedule long-running tasks asynchronously.
Astronomer is a finalist for the 15th Annual Ventana Research Digital Innovation Awards, recognized for innovative technologies in their markets.
Discover Astro – the data orchestration platform powered by Airflow. Find out how to build, run, and observe data pipelines efficiently and with context.
Learn how the new Astro CLI commands provide a great DAG development experience for Airflow users. Get to know the dev parse and dev pytest commands.
Hear some reasons organizations consider building their own Apache Airflow® infrastructures, and how a fully managed service makes you more competitive.
Find out more about Astronomer Providers, a set of Airflow 2-licensed providers with async functionality, created and maintained by Astronomer experts.
Learn what’s new in the latest release of Apache Airflow® 2.3 and how it can improve data orchestration.
Learn all about the role and challenges of a data scientist and find out how Apache Airflow® can help with your workflows.
Learn best practices for standing up, scaling, and growing Apache Airflow® to support modern data orchestration.
Learn how to use Airflow and dbt together to advance data orchestration and data transformation projects and facilitate collaboration across data teams.
Discover the power of data lineage and its role in improving data observability and quality. Find out how to take data orchestration to the next level.
Learn how Astronomer acquired Datakin, the real-time, operational data lineage tool from the founders of the OpenLineage and Marquez open-source projects.
Hear how Joe Otto reflects on Astronomer’s history, and looks to a future powered by the combination of orchestration, lineage, and observability.
The site covered our recent acquisition of Datakin and our Series C round.
Learn the common challenges data and analytics leaders face and how they use Apache Airflow® and Astronomer to empower themselves and their data teams.
The biggest community-driven event around Apache Airflow® returns May 23–27, 2022.
Learn how Astronomer became the top data orchestration platform based on Apache Airflow®. See how to apply Airflow in your ETL and analytics use cases.
Master Apache Airflow® with these 10 best practices. Learn how to optimize your data pipelines, improve efficiency, and avoid common pitfalls.
Learn about emerging trends that are revolutionizing the world of data from the leading Apache Airflow® experts. See how to efficiently manage data in 2022.
Find out how to use Great Expectations in an Airflow Directed Acyclic Graph to successfully perform, prioritize, and schedule data quality checks.
Learn more about Astronomer’s new partnership with Uturn Data Solutions – the leading experts in enterprise cloud enablement and application modernization.
Gain insight into the role and responsibilities of a data engineer. See 7 examples of how Apache Airflow® can make data engineering less challenging.
Learn how to orchestrate Azure Data Explorer queries with Airflow.
Learn from our experts about how to select the best ETL tool, and why it pays to integrate Fivetran, Airbyte, and Azure Data Factory with Apache Airflow®.
Hear from Bolke de Bruin – VP of Enterprise Data Services at Astronomer – about how Apache Airflow® helps modern companies manage data effectively.
Learn about machine learning orchestration, machine learning pipelines, and their components. See why Apache Airflow® is the top ML data orchestration tool.
Breaking down what a modern data stack means in practice. We discuss four core components, five reasons to set it up, and how to orchestrate it.
Explore the differences and similarities between Apache Beam and Airflow. Understand their capabilities, programming models, and ideal use cases to make the right choice for your data management needs.
See why modern orchestrators and reverse ETL tools are the future for data-driven business. Learn how Apache Airflow® takes SQL to the next level.
Discover what a machine learning pipeline is and the process behind creating one with Apache Airflow®. Learn what you need to know about ML pipelines.
Understand the difference between ETL and reverse ETL. Learn to use the Census reverse ETL platform and Airflow together to leverage data orchestration.
Hear what the BBC's Data Engineer says about the popularity of orchestration tools in the media industry. Find out why the BBC went for Apache Airflow®.
See what’s new in Apache Airflow® 2.2. Learn about big improvements, bug fixes, and internal changes, and the benefits they bring to Apache Airflow® users.
Learn more about the concept of Big Data Architecture and its 5 core components. See how various companies benefit from implementing Big Data Architecture.
Learn how to make the most of data in banking. See which banks use Airflow, and how the top orchestration tool helps them overcome major challenges.
Get to know the major benefits and limitations of Apache NiFi and Apache Airflow, and see which of the two popular ETL tools is better for data management.
Alexandra Abbas—a Machine Learning Engineer at Wise—explains what makes Airflow an ideal tool for data orchestration in the fintech industry.
Learn what an ETL process is and how to build it. Find out how Apache Airflow® can help you create, scale, and manage ETL pipelines more effectively.
Find out what causes data silos and how they hurt your business. Learn how Airflow and data orchestration can solve the data silo problem in your company.
Hear from the Product Owner at Societe Generale about the benefits of implementing Airflow. Find out how data orchestration solutions are used in banking.
Learn the basics of data pipeline building. Get to know data pipeline components, types, and best practices. See how Airflow can simplify the process.
Streamline DAG generation! Explore a utility package to create Airflow DAGs from dbt models. Get sample configurations & customizable code.
Data orchestration is the process of collecting siloed data from multiple locations and systemizing, unifying, and activating it for data analysis.
Learn what the Airflow community got up to in 2021, in this recap of the biggest international Airflow event. Get ready for the next Airflow Summit!
Hear from Viraj Parekh–Field CTO at Astronomer–about how data pipelines help increase online sales. See why Apache Airflow® works for e-commerce companies.
Learn why it’s worth attending the biggest Airflow conference for developers and data professionals. Check what’s on the agenda and register for free.
Find out how to get started with Apache Airflow® and enhance your knowledge. Learn essentials from Astronomer’s experts and become Apache Airflow®-certified!
Learn more about the KubernetesExecutor and its upgrade to version 2.0. See new features redesigned with Airflow admins and data engineers in mind.
Learn how to orchestrate Talend jobs with Airflow so you can use both tools without rewriting your pipelines.
Learn about the discovery-and-distribution hub for Airflow integrations. See how to bridge the gap between the Airflow community and the data ecosystem.
Learn more about the TaskFlow API and read about its features. Get to know how TaskFlow API in Airflow 2.0 enables a better DAG authoring experience.
Have a look at Astronomer’s ultimate guide on Airflow Secrets, and learn best practices for managing Secrets with various backends in Apache Airflow® 2.0.
Learn how to implement near-real-time Change Data Capture (CDC) in Airflow using a scheduled GCP CloudSQL export approach for data pipelines.
Take your Airflow DAGs live! Learn how to deploy them in production using dbt manifest.json, and integrate dbt into your ETL/ELT workflows.
Kickstart your analytics architecture with Airflow and dbt. Learn DAG authoring, configurations, and code snippets for a seamless setup.
Explore the features of the updated Apache Airflow® 2.0 Scheduler. Learn how the Airflow Scheduler enables quick and seamless initiation of tasks.
Find out why Great Expectations and Apache Airflow® are a great match. Learn how to leverage native Great Expectations functionality directly from DAGs.
Get to know the highlights of Apache Airflow® 2.0 and see hundreds of new features it includes. Have a look at how Airflow 2.0 compares to Airflow 1.10.
Explore the possibilities of the Kubernetes Event-Driven Autoscaler. See how KEDA helps users improve the efficiency of their Apache Airflow® deployments.
Get tips on improving the performance and reliability of the Airflow Scheduler. Find out how to benchmark and profile it using py-spy and Flame Graphs.
Discover Apache Airflow® and explore its workflow-management capabilities. See which global companies use Airflow to solve data engineering challenges.
Learn how Astronomer Cloud supports the latest version of Apache Airflow®. See the features included in the newly released, next-generation data platform.
Learn more about Astronomer v0.10 and its key updated functionalities. See how the new Astronomer Platform supports the latest version of Apache Airflow®.
Get to know best practices for debugging Apache Airflow® DAGs. Check out the list of common Airflow deployment errors, and see how to find and remove them.
Find out how to design an Airflow infrastructure, and whether it makes more sense to power your DAGs with one monolithic Airflow instance, or many.
Discover the newly launched Astronomer v0.8.0 and the features it includes. Find out what’s been fixed, improved, and added to the Astronomer Platform.
Get to know why Astronomer switched to Apache Airflow®. Learn how we optimized Airflow to fit our initial needs and what features we're planning to build.
Release notes covering the features released with v0.7.0 of the Astronomer platform.
Release notes for v0.6.0 of the Astronomer platform.
Release notes from our recent platform update to v0.5.0.
Check out the highlights of the Astronomer v0.4.1 release. See the full summary of upgrades and learn more about the Astronomer Platform's new features
Explore the future of Apache Airflow® at Astronomer. Take a look at the official Airflow roadmap and see what improvements and developments to expect.
Learn more about Astronomer v0.3.2 and its new, updated functionalities. Find out what’s been changed, fixed, and added to the Astronomer Platform.
Find out more about Astronomer v0.3 and its great benefits. Get to know what features are included in the newly released next-generation data platform.
Learn about the benefits of the Astronomer’s Managed Service for Apache Airflow®. See why our experts decided to design and build the Astronomer Platform.
Discover Astronomer SpaceCamp and see how it gets data teams up and running with Airflow in no time. See the benefits of different SpaceCamp versions.
Learn about Astronomer's podcast focused on the future potential of Apache Airflow®, as seen by top players in the data engineering space.
Check out instructions on importing and moving Github data using Apache Airflow®. See how to deal with Github DAG writing, visualization, and dashboarding.
Hear from Maksim Pecherskiy–Chief Data Officer of the City of San Diego–about how Apache Airflow® helps operationalize data in the public sector.
Find out how to scale behavioral analytics with Apache Airflow®. See why the Astronomer Platform is an ideal solution for data scientists and analysts.
Learn why ARGO chose Apache Airflow® to build and maintain data infrastructure. Find out how they transform basic public services with Airflow.
Explore common data format types (CSV, JSON, XML) & understand their pros, cons, & ideal use cases. Learn how to choose the right format for your data needs.
Get to know the roles and responsibilities of data scientists and data engineers. Learn why data engineering and data science go hand in hand.
Learn what a DAG is and how it's used in data pipelines. Explore benefits, real-world examples, and FAQs in this comprehensive guide.
Learn how Astronomer, a data engineering platform, closed $3.5M in new financing, led by Wireframe Ventures in San Francisco and CincyTech in Cincinnati.
Knowing the options for storing data will help you make the right decisions for your company when you’re ready to take this step.
Hear from Maxime Beauchemin – a data engineer at Airbnb and creator of their data pipeline framework, Airflow – about the future of data engineering.
Get to know how the open-source approach helps drive growth and innovation. Learn why it’s worth investing in open-source components like Apache Airflow®.
Learn more about the different types and properties of hard-to-reach data with great potential. Find out how to access, organize, and store it effectively.
Learn why Astronomer needed a unified scheduling system to extract and monitor all types of data pipelines. Find out why Apache Airflow® was our answer.
Press Release: Astronomer Closes $1.9M in Seed Financing
Learn about Astronomer’s 2-year journey to raising $2 million seed, including the ups and downs we went through to take our company to the next level.
See how to simplify the data pipeline writing process with the right tools. Learn what Astronomer experts do to make data pipelines less challenging.
Find out how Astronomer leveraged AWS services with open-source alternatives. See why Airflow and Apache Mesos help build more and better integrations.
Coming out of AngelPad’s 2015 Demo Day, we found ourselves vacillating between an acquisition and Series A, though we were arguably too early for either.
Release notes for v0.9 of the Astronomer Platform
Astronomer's Head of Design, Chris Hendrixson, explains how he created the design aesthetic to encompass data, futurism, and a little bit of fun.
Redshift is popular but you still need to know what you''re doing when spinning up your first cluster. In this tutorial, we walk you through the process.
Learn how to leverage your business data with data warehousing. Discover the best time to create a data warehouse, and see which warehousing tools to use.
Find out why it was worth driving from Cincinnati to New York for a fifteen-minute meeting with the number one ranked startup accelerator in the world.
Disaster Recovery on Astro is Generally Available. A look at the architecture decisions, trade-offs, and engineering lessons behind cross-region Disaster Recovery on Astro - from database replication with Aurora Global Clusters to warm standby compute and one-click failover
How the Astronomer Education team uses AI to help respond to user tickets and answer questions about Academy usage.
How Astronomer built an AI agent that reads every Apache Airflow commit daily to catch breaking changes before they reach customers.
How Astronomer built Kepler, an internal Slackbot and CLI tool that answers ad hoc data questions using LLMs, hybrid search, and code context.
How a Principal Product Manager used AI coding agents like Claude Code during onboarding to automate workflows, track customer conversations, and ship features in his first 90 days at Astronomer.
One year ago, we rolled out a new incident management process. Read on to find the past, present, and future of incident management at Astronomer.
Discover how we created an efficient release note system with Towncrier. Minimized friction and improved developer workflows.
A walkthrough of Astro's architecture and deployment model
OR
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.