How We Built Cross-Region Disaster Recovery for Astro on Google Cloud
How we built it Cross-region disaster recovery on GCP: the architecture, the decisions specific to Google Cloud, and what teams need to plan before starting.
Astronomer Blog
The latest insights from our team of Apache Airflow® experts.
Debunking myths about Airflow’s use cases
Learn the truth about time-based scheduling limitations, ML/AI workflow compatibility, streaming processing, and whether Airflow is outdated technology.
Debunking myths about the Airflow user experience
Debunking some common Apache Airflow misconceptions.
Debunking myths about Airflow’s architecture and performance
In this second post, we’ll turn our attention to another common theme in discussions about Airflow: its architecture and performance.
A redesigned Astro for teams managing at scale: one view across every workspace, deployment, and pipeline
Built for teams managing multiple workspaces, Astro's redesigned navigation puts your entire org in one view the moment you log in.
Introducing Apache Airflow® 3.3
Better failure handling with the task state store, pluggable retries, and more
The Airflow Build vs. Buy Question We See Data Leaders Get Wrong
Apache Airflow is free to run. What it costs to run well is a different question. Here's how to think through the build vs. buy decision — and what most data leaders get wrong.
From Author to Review: Otto Now Covers the Full Dag Lifecycle
Otto runs automatically on every pull request, reviewing your Dag code before a human reviewer ever opens the diff.
Investigate Every Pipeline Failure Automatically With Otto
Otto investigates critical failures before you open your laptop
Introducing AI-Assisted Migrations with Otto: Your Airflow Expert for the Translation Work that Stalls Every Project
Otto by Astronomer uses AI to convert Control-M, AutoSys, and Automic (UC4) job definitions into production-ready Airflow Dags. Faster migrations, less technical debt.
Airflow in Action: ETL Insights from Bloomberg — Slashing Runtimes by 50%
Bloomberg's Data Platform Engineering team shares four real-world Airflow configuration fixes that resolved production issues across 2,500+ DAGs — no code changes required.
Why Astronomer Support Is Chasing Michelin Star Service
Astronomer’s Customer Reliability Engineering team was recently honored to receive an award by Business Intelligence Group for our support, a step toward our audacious goal of being the best software support team in the world. We are proud to work with such an incredible customer base, many of whom use Astro for mission-critical workloads. That responsibility demands an uncommon level of service and excellence.
Astro Private Cloud 2.0: Disaster Recovery, Governance, and Audit Logging on Your Terms
Astro Private Cloud 2.0 in Generally Available
Airflow in Action: Migrating 200,000 Pipelines to Airflow 3 at Uber
Uber is migrating 200k pipelines to Airflow 3. Here's the architecture, tooling, and strategy behind one of the largest orchestration migrations ever undertaken.
Introducing Otto: The Only Data Engineering Agent Built for Airflow
Otto is the only data engineering agent purpose-built for Airflow, and the first to bring Astronomer''s depth of operational Airflow knowledge directly to the engineers doing the work.
Migrating from airflow-ai-sdk to Apache Airflow's Common AI provider
apache-airflow-providers-common-ai 0.1.0 shipped in April 2026, Airflow's official provider for AI and LLM workflows, built on PydanticAI and sharing its foundations with the airflow-ai-sdk. Same decorators and engine, plus toolsets that turn 350+ provider hooks into agent tools, durable execution, and built-in human review. Here's what's new, why it matters, and how to migrate without losing a weekend.
Cross-Region Disaster Recovery on Astro Is Now Generally Available: Here's How We Built It
Disaster Recovery on Astro is Generally Available. A look at the architecture decisions, trade-offs, and engineering lessons behind cross-region Disaster Recovery on Astro - from database replication with Aurora Global Clusters to warm standby compute and one-click failover
AI Agents Have a Context Problem. It's Hiding in Your Pipelines.
The richest source of context for AI agents already exists in your stack. It's your orchestration layer. Here's why static metadata alone won't get you there.
Cosmos 1.14: Battle-Tested Watcher Mode
Cosmos 1.14 delivers significant improvements to Watcher execution mode, the fastest way to run dbt in Airflow, plus a fully restructured documentation experience built around how teams actually use Cosmos in production.
Airflow in Action: Kaiser Permanente's AI Pipelines Detecting Disease Before Clinicians Can
See how Kaiser Permanente runs AI and ML research pipelines on-prem with Airflow, and how Astronomer supports regulated industries with the same flexibility
Airflow in Action: How SAP Delivers Trusted AI for Enterprise Clients
See how SAP built a production RAG pipeline with Apache Airflow to power Joule for Consultants, processing 5M+ documents across 15 data sources for the enterprise.
Introducing Blueprint in Astro: Self-Service Dag Authoring For Your Entire Organization
With Blueprint, creating Airflow pipelines is now possible for anyone in your organization. Data engineers define templates, and others can create pipelines though a drag-and-drop no-code interface in Astro. No python or Airflow knowledge required.
How Blueprint and Astro IDE Redefine Orchestration
Learn how Blueprint and the Astro IDE let platform teams define governed templates so anyone can build Airflow pipelines without writing Python.
Introducing Apache Airflow® 3.2
Asset partitions, async tasks, and continued improvements to the Airflow 3 platform
The EduAgent: The AI Agent Behind Astronomer Academy's Support
How the Astronomer Education team uses AI to help respond to user tickets and answer questions about Academy usage.
Airflow in Action: From Steel City to Data City. Pittsburgh, Astro and Open Government Data
See how Pittsburgh's two-person data team uses Apache Airflow and Astro to power open government data, cut manual work, and deliver real civic impact.
The AI Agent Behind Astro Runtime's Reliability
How Astronomer built an AI agent that reads every Apache Airflow commit daily to catch breaking changes before they reach customers.
Airflow in Action: How DoorDash Scaled for Data and ML Engineering
DoorDash runs one of the largest Airflow deployments in the world. See how they built the Orchestration Frederator to scale without the chaos.
Airflow in Action: Red Hat's Blueprint for Trusted AI Agents with Astro
How Red Hat built a trusted data and AI platform on Apache Airflow and Astro, powering 95 data products and production AI agents across the enterprise.
Build Faster, Let Agents Do More: What's New in the Astro CLI
The Astro CLI is the fastest way to develop and deploy Airflow pipelines from your own editor. Today we're making it more powerful with standalone mode, automatic port management, and direct API access.
Airflow in Action: How American Express Orchestrates Metadata Across 3,000 Databases
Amex runs 100+ production Dags orchestrating metadata with Airflow. See how they built the operators, pipelines, and governance framework that makes it work.
Cross-Region Disaster Recovery on Astro: Enterprise Resilience, Without the Engineering Project
Today, we are announcing the public preview of cross-region Disaster Recovery on Astro, now available on AWS.
Improving Reliability with a More Resilient Auth Proxy Architecture
How Astronomer's new dataplane-based auth architecture improves Astro resilience, reduces blast radius, and keeps DAG runs running during control plane outages.
Build Your Data Quality Deck
Master Apache Airflow's six SQL check operators to build comprehensive data quality checks. Learn when to use each operator and how to sequence them effectively.
Airflow in Action: Boosting BI at Visa. 100% Trust in Data With 70% Lower Overhead
Visa cut BI dashboard refresh times from 24 hours to 2 hours with Airflow. Learn how they automated end-to-end workflows and achieved zero missed SLAs.
Building Kepler
How Astronomer built Kepler, an internal Slackbot and CLI tool that answers ad hoc data questions using LLMs, hybrid search, and code context.
What Happens When a PM Runs a Coding Agent (almost) Full-Time
How a Principal Product Manager used AI coding agents like Claude Code during onboarding to automate workflows, track customer conversations, and ship features in his first 90 days at Astronomer.
Airflow in Action: Unlocking Airflow 3's Power for Multi-Tenancy at Datadog
Datadog processes 100+ trillion events daily. Learn why they chose Airflow 3 for multi-tenancy, and data-aware scheduling at scale.
Dag-Level Roles on Astro: Fine-Grained Access for Enterprise Airflow
Dag-level roles are now available on Astro, giving Enterprise customers the ability to control access to individual Dags within a shared deployment.
Airflow in Action: Best Practices Learned From Scaling AI at Oracle
Best practices from AI at Oracle: use Airflow to scale GPU-heavy AI and MLOps on Kubernetes with higher reliability, visibility, and GPU ROI.
Airflow in Action: Expedia’s Multi-Tenant Platform at 200+ Clusters, 14,000+ Pipelines
How Expedia runs 14,000+ Dags across 200+ isolated Airflow clusters with multi-tenancy, templates, and automated CI/CD.
Introducing AI Agent Tooling: Bringing Airflow Intelligence to Your Local Workflow
Whether you use Claude Code, Cursor, VS Code, or any of the 25+ compatible AI coding tools, you can now access specialized Airflow knowledge right where you already work.
How to build a decision tracing context graph with Apache Airflow®
Learn how to build a decision-tracing context graph with Apache Airflow using the HITLOperator, Airflow AI SDK, and a Slack plugin to capture human decisions.
The 2026 Astronomer Data Excellence Awards
This year’s winners span industries, regions, and use cases, but they share a common thread: data orchestration as a strategic advantage.
Airflow in Action: Qualcomm Chip Design Orchestration at Million-Task Scale with Airflow 3
How Qualcomm uses Airflow 3 to orchestrate chip design at massive scale, running millions of EDA tasks daily across global data centers.
Introducing Astro API: Production-Ready and Generally Available
Whether you're building new automation or migrating from the beta API, this is the stable foundation you need to programmatically manage Astro at scale.
Airflow in Action: Powering Investigative Journalism at the FT with Orchestration and AI
How the Financial Times uses Apache Airflow to orchestrate AI pipelines, structure messy public data, and power investigative journalism at scale.
Astro Observe Data Quality: Now in Public Preview with Event-Driven Monitoring
Data Quality monitoring in Astro Observe is now available in public preview with powerful new capabilities that fundamentally change when and how you validate data.
State of Airflow 2026: The Orchestration Layer is Uniting Data, AI, and Enterprise Growth
We've just released the State of Airflow 2026 report, built on insights from over 5,800 data professionals across 122 countries—the largest survey of data engineers ever conducted.
Introducing Cohort 6 of the Astronomer Champions Program for Apache Airflow®
This new cohort brings together an outstanding group of data engineers, architects, and platform leaders from around the world who share a deep commitment to Apache Airflow® and to advancing the practice of data orchestration.
Building Data Pipelines Like Assembly Lines
Scale your data engineering team with a declarative Airflow framework and the write-audit-publish pattern, making pipeline development fast, safe, and reliable.
Localized Orchestrators Will Lock You Out of Enterprise AI
There’s an urgent reason why localized orchestrators are a strategic mistake. One that has nothing to do with exit costs or vendor leverage and everything to do with what AI needs to work.
Airflow in Action: Orchestrating AI At Duolingo To Unlock 99% Lower Costs, 10x User Growth
Discover how Duolingo uses Airflow to orchestrate AI content pipelines, automate critical workflows, cut costs, and scale learning to millions.
Airflow in Action: Inside Deutsche Bank’s Regulated Data Workflows
Learn how Deutsche Bank runs Apache Airflow in a highly regulated environment to orchestrate audited, hybrid banking workflows at scale.
Upgrading Airflow 2 to Airflow 3 - A Checklist for 2026
You’ll learn: The 3 big reasons for upgrading to Airflow 3, how to plan your Airflow upgrade, and more.
Better together: Astro CLI and Astro IDE
Astro CLI and Astro IDE serve different feedback loops. Learn when to use each—and how combining them helps teams build and ship Airflow faster.
Data products get budgets, pipelines get questioned
By treating your critical pipelines as products, you make the invisible visible, and you might just get the budget (and the sleep) you deserve.
What's New in Astro IDE: Fast Mode and Enhanced Skills
We’ve made Astro IDE even more powerful, with a set of improvements providing the agent with more context, and data engineers more control over how the agent responds.
What is Data Observability?
A guide to understanding data observability, why it matters, and how modern teams are using it to deliver trusted, reliable data at scale.
Airflow in Action: Inside GitHub’s Data Platform. Open Source to Copilot
How GitHub uses Airflow for open source insights, customer success and continuous Copilot improvements plus nine years of lessons from its data platform.
Airflow in Action: The Unified Orchestration Platform Behind OpenAI
How OpenAI scaled Apache Airflow into a unified orchestration platform, boosted reliability, and replaced legacy schedulers across the business.
Best practices for writing Airflow Dags with AI
Learn how to effectively use AI tools like the Astro IDE to write maintainable Airflow DAGs by providing proper context, using structured prompting techniques, and enforcing governance standards rather than relying on "vibe coding."
Multi-Agent orchestration with Apache Airflow®, Apache Kafka®, Aryn AI, and OpenAI
Build multi-agent AI pipelines with Airflow, orchestrating specialized agents, RAG with Aryn AI and Weaviate, human-in-the-loop steps, and Kafka.
Cut DAG Runtimes by Up to 80% with Cosmos Watcher Execution Mode
This release focuses on helping teams run dbt workflows faster while maintaining the task-level observability that makes using Cosmos superior to running dbt projects with the BashOperator.
Abstraction with DAG Factory: From Excel to Minecraft
In this second part, we explore the middle ground between full code and full abstraction: configuration-based authoring with DAG Factory.
From Reactive to Proactive: How Astro Observe Helps Airflow Teams Prevent Alert Fatigue and Catch Failures at the Source
Teams can move from reactive alerts to proactive prevention, detecting issues at the source before they break dashboards or delay critical SLAs.
Speedrun your first Apache Airflow® Dag run
The Astro IDE allows you to run your first Airflow Dag within X minutes! Can you beat my time?
Calculating Airflow’s Total Cost of Ownership
Model the true cost of Airflow. Use Astronomer’s TCO Calculator to uncover savings and ROI from a fully managed, unified orchestration platform.
The Rise of Abstraction in Data Orchestration
We look at how abstraction reshapes orchestration, why it matters for technical managers, and how to position teams to thrive in a multi-level reality.
Introducing Astro Private Cloud: An Enterprise Orchestration & Scheduling Platform in Your Environment
Astro Private Cloud delivers enterprise-grade data orchestration with control plane and data plane separation, enhanced security, and Apache Airflow 3 support.
Astro IDE and Human-in-the-Loop: A First Look at Airflow 3.1 and AI Agent comparison
As part of the Astronomer DevRel team I get to explore new features in the weeks before the release to produce educational content, for humans but also for our friendly helpful robots: AI coding assistants.
Introducing Apache Airflow® 3.1
The momentum continues from the release of Airflow 3
Astro IDE in Action: How Data Teams Are Accelerating With AI
Since launching in private preview earlier this year and over the course of the public preview, design partners across industries – from automotive to travel, logistics, education, and sports – have been putting Astro IDE to work.
Meet the Astro Executor: Faster DAGs, Fewer Failures, Lower Costs
Introducing the Astro Executor, the Airflow 3 architecture built for performance and reliability. Get 70% higher concurrency, fewer failures, and lower infrastructure costs.
Orchestrating Data Quality with Airflow
Here we dive into why + how the Data Team built scalable, maintainable data quality checks inside the developer experience and made maintaining high data quality an attainable outcome.
Why Observability Belongs in Your Orchestration Platform
Challenges of multi-tool observability, Airflow's limited features, and how Astro Observe offers comprehensive pipeline monitoring in one platform.
Introducing Astro IDE: Ship Apache Airflow Dags 10x Faster
Today we're announcing Astro IDE, the first AI-powered IDE purpose-built for Apache Airflow.
DAG Factory 1.0: Simplifying Airflow DAG Creation for Modern Data Teams
DAG Factory, the open-source tool for declarative DAG authoring in Apache Airflow, reaches a major milestone with version 1.0.
The Road to Reliability: Implementing Pipeline Observability
Our team has been focused on pipeline reliability and operational confidence. We'll share our journey: where we started, where we struggled, and where we are now.
Meet the Astronomer Data Team
We’re Astronomer’s in-house data team. We center around two main goals: make data valuable and reliable at Astronomer , all with Airflow!
Join us for Beyond Analytics, coming September 16
Get your free ticket to explore how data engineering is evolving from internal analytics to mission-critical applications!
Supercharge dbt Orchestration with Astronomer Cosmos and Apache Airflow®
Cosmos 1.11.0a1 introduces alpha support for dbt Fusion—the next-generation dbt engine that unlocks lightning-fast parsing, state-aware orchestration, and real-time validation, all orchestrated natively in Airflow.
In an AI world, it’s the workflow that allows you to build your moat
While LM Arena leaderboards grab headlines, an infrastructure battle is happening in the orchestration systems that transform AI from lab experiments into production solutions that solve real-world problems.
Airflow in Action: Modernizing Legacy Data Systems for AI and Analytics With Procter & Gamble
Discover how Procter & Gamble uses Apache Airflow to modernize data systems, power AI, and streamline legacy workflows at global scale.
Introducing Data Quality in Astro Observe: The Orchestration-First Approach to Reliable Data
Astronomer is excited to announce enhanced data quality monitoring in Astro Observe, now available in private preview.
Introducing Cohort 5 2025 of the Astronomer Champions Program for Apache Airflow®!
Meet Cohort 5 of the Astronomer Champions Program for Apache Airflow in 2025. Data leaders from top global organizations are driving data engineering's future.
Secure, Flexible Data Orchestration: Meet Remote Execution on Astro
With Remote Execution on Astro, enterprises no longer have to choose; you can now run workloads exactly wherever they need to be, while still benefiting from centralized orchestration and observability in Astro.
Why Enterprise AI Struggles: The Context Gap, Data Gravity, and What Comes Next
Discover why context is vital for effective AI implementation and how shifting from centralized data systems to context-rich approaches enhances decision-making.
Introducing Apache Airflow® 3: the most significant release in Airflow's history
Reimagining data orchestration for the AI-era
Native support for Source Node Rendering in Cosmos
Cosmos now natively supports source node rendering! Visualize your dbt DAG more effectively with this enhanced feature. Learn how it simplifies data lineage.
The Operating System for Enterprise AI
In the AI era, orchestration platforms become the lynchpin.
From Regulation to Resilience: How Astronomer Powers DORA-Ready Data Orchestration
The Digital Operational Resilience Act (DORA) addresses the critical need for the financial sector to effectively manage digital operational resilience in the face of rising cyber threats and Information and Communication Technology (ICT) disruptions.
Workflows then agents: the practical approach to enterprise AI
Introducing the AI SDK for Apache Airflow™
Enhanced Alerting at Astronomer: Streamlined Airflow Monitoring at Scale
Astro Enhanced Alerting provides a unified monitoring experience through pattern-based alert rules that teams can apply seamlessly across hundreds of DAGs simultaneously.
Astronomer and IBM Collaborate to Transform Enterprise Data Orchestration
Astronomer and IBM join forces to revolutionize enterprise data orchestration! Discover how this exciting collaboration will transform data management.
Airflow in Action: How Vibrant Planet Accelerates Climate Resilience With Geospatial Analytics and ML
Vibrant Planet optimizes Airflow pipelines for wildfire prevention, using ML and dynamic memory scaling to process massive geospatial datasets
Introducing new flexible billing options for Astro
You can now upgrade to our Team plan while continuing to only pay for what you use, billed monthly to a credit card or cloud marketplace.
Apache Airflow® 3 Development Update
Airflow 3 is the biggest release in Airflow’s history, the result of a massive effort from the global Airflow community.
State of Airflow 2025: Unleashing the Future of Data Orchestration
Key Findings from the State of Airflow Report 2025
Airflow in Action: Scaling Climate Intelligence with Zero-Downtime on HPC Clusters at Meteosim
Integrating Slurm and Airflow, Meteosim runs over 6,000 pipelines daily with zero failures. Read the post to learn how they did it.
Airflow in Action: Customizing LLMs at Laurel. Speeds Rollout and Saves $500k / Year in Inference Through GenAI Ops
How Laurel uses Airflow to scale GenAI, automate timekeeping, and cut LLM costs by $500K per year. Learn how DAGs power model retraining and inference.
Astro Observe is Now Generally Available
The new standard for pipeline reliability and data product observability.
FreightWaves scales Data Engineering with Astronomer and Apache Airflow
FreightWaves transitioned from a scattered and inefficient setup to Astronomer’s Astro, powered by Apache Airflow, to transform their data engineering practices.
Airflow in Action: Insights From Scaling Multiple Workloads Across A Shared Environment at Rakuten Kobo
Discover Rakuten Kobo’s strategies for scaling Airflow usage across teams, fostering collaboration with guardrails, alerts, and shared environments.
Why Orchestration and DataOps Will Redefine the Modern Data Stack
DataOps is the future of the modern data stack. Learn how orchestration-first platforms like Astro are unifying the stack & delivering value.
Celebrating Innovators in Data Orchestration: the Astronomer Data Excellence Awards
Discover the winners of the Astronomer Data Excellence Awards! Celebrating groundbreaking achievements in data orchestration with Astro and Airflow.
Airflow in Action: Making dbt on Airflow Easy with Astronomer Cosmos. Insights from BAM
BAM simplifies complex data pipelines with dbt on Airflow using Cosmos, enabling teams to scale transformations with ease and transparency.
Modernizing legacy ETL workloads with Airflow
How to Migrate Legacy ETL Workloads from Informatica to Airflow DAGs Using DAG Factory
Airflow in Action: 33% Faster Deploys with 90% Higher Data Quality. Astronomer at Autodesk
Discover Autodesk’s secure UAT environment for PII testing, achieving faster deployments and reduced technical debt, powered by Airflow and Astronomer.
TrackFly Optimizes Data Workflows with Astronomer and Apache Airflow
TrackFly turned to Astronomer’s managed Airflow solution, Astro, to streamline their ETL (Extract, Transform, Load) workflows and focus on core business goals.
Airflow in Action: 5.5K Pipelines, 200 Models, 10% of the World’s Web Sites. MLOps Insights From Wix
Wix runs 5,500 Airflow pipelines daily, powering ML workflows for the company that hosts 10% of the web. Explore their innovative MLOps platform.
Airflow in Action: Deploying AI Clusters to 100 Data Centers in 3 Months. Infrastructure Insights from Cloudflare
Learn how Airflow powers Cloudflare’s infrastructure automation, from server diagnostics to GPU inference clusters in 100+ data centers
Airflow in Action: Data Engineering Insights from Circle, Managing $10s of Billions Across Multiple Blockchains
Learn how Circle manages stablecoins across multiple blockchains with Airflow, tackling MWAA challenges and exploring Astronomer for speed & scale
Airflow in Action: $160BN in Assets, 4,000 Data Pipelines. Financial Workflow Insights from Robinhood
Learn how Robinhood scaled mission-critical financial workflows with Airflow, from trading to clearing, migration strategies, and future plans.
Airflow in Action: Orchestration Insights from 2,200 Pipelines at Instacart
Discover how Instacart scaled Apache Airflow to handle 2,200 pipelines and 16 million tasks monthly with custom tools, IaC, and centralized management.
Airflow in Action: ML and Data Engineering Insights from the Analytics-Obsessed World of MLB
Explore how the Texas Rangers and Philadelphia Phillies use Airflow to revolutionize data orchestration and ML, driving faster insights and a competitive edge.
Data Observability 101: An Introduction to the Most Critical Features of Modern Data Observability
Explore core data observability features: analytics, monitoring, and alerts. Gain insights into pipeline health, prevent failures, and ensure data reliability.
Airflow in Action: How Panasonic Energy is Accelerating the EV Transition With Data Engineering
Learn how Panasonic leverages Apache Airflow to streamline battery production, scale workflows, and plan for streaming and Kubernetes.
ELT for Beginners: Extract from S3, Load to Databricks and Run Transformations
Learn how to build an ELT pipeline extracting data from S3, loading it into Databricks, and transforming it with notebooks using Apache Airflow.
Airflow in Action: ML and LLM Ops Insights from ASAPP — Reducing Workflow Runtimes by 85%
Discover how ASAPP turbocharges MLOps with Airflow and Spark, scaling LLM workflows to boost efficiency & achieve lightning-fast processing in hours vs days.
Airflow in Action: Infrastructure Management Insights from 1 Million Monthly Deploys at LinkedIn
Discover how LinkedIn uses Airflow to orchestrate 12k pipelines managing 1m deploys for 7k services. And Airflow 3 will make provisioning even easier!
Airflow in Action: Data Engineering Insights from Processing PBs of Data Every Day at Stripe
Learn how Stripe processes petabytes of data daily with Apache Airflow, ensuring compliance while accelerating developer workflows with its dev/test tooling
Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day
Go on an Airflow journey with Bosch as they scale from 1k to 50k DAG runs per hour and 1.2 million pipeline runs per day with an average 1 second latency.
Customer Story: Autodesk’s Data Engineering Transformation with Astronomer and Apache Airflow
Autodesk scales with Astro! 🚀 Learn how they migrated from Oozie to Airflow, boosting efficiency, scalability, and data-driven decision making.
It Pays to be Picky: Leveraging Key Insights with Airflow and Astro
Here are four key observability insights to consider focusing on when setting up an observability solution: data freshness, on-time delivery, data dependencies tracking, and data quality.
Airflow in Action: ETL Insights from Bloomberg — Slashing Runtimes by 50%
Learn why Bloomberg selected Airflow over Dagster and Prefect, reducing the ETL of 50 million loans and 5 billion data points by 50%
Airflow in Action: Data Engineering Insights from Burns & McDonnell
Discover how Burns & McDonnell scaled data delivery from ad-hoc workflows to a unified platform using Apache Airflow, providing reliable data in under 24 hours.
Update: Astro Observe is Now In Public Preview
Today, we’re excited to announce Astro Observe, which brings new and more robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC).
Proactive Airflow Monitoring: How to Prevent Infrastructure Issues Before They Happen
Prevent Airflow downtime with proactive deployment health alerts. Automate monitoring & get actionable insights for reliable data pipelines.
Airflow in Action: Data Engineering Insights from Uber and Its 200,000 Data Pipelines
Discover how Uber streamlined data workflows with Apache Airflow, scaling to support 1,000 teams, 450,000 daily pipeline runs, and plans for Airflow 3
Unlocking the Power of Scalable Machine Learning with Anyscale and Astronomer
Anyscale and Astronomer join forces to solve the challenges of scaling machine learning and AI.
Introducing Cohort 4 of the Astronomer Champions Program for Apache Airflow!
This final cohort for the 2024 calendar year brings together data leaders from Fortune 500 companies around the globe, all possessing extensive Airflow expertise.
Announcing Astronomers Vulnerability Disclosure Program
Astronomer launches Vulnerability Disclosure Program, a way for us to better engage with the global community in receiving, recognizing, and rewarding findings from the collective security community.
Best Practices and Solutions for Multi-Tenant Airflow
How Isolated Environments Enhance Execution, Security, and Productivit
Introducing Cosmos 1.6: The Best Way to Run dbt-core in Airflow
We're happy to announce that the newest version of Astronomer’s dbt-core integration, Cosmos 1.6.0, is now available, featuring a list of enhancements and some great new additions to serve the community.
Astro and Terraform: Empowering Infrastructure as Code for Modern Data Orchestration
Announcing the Astro Terraform Provider! Use Terraform to manage and automate your Astro deployments.
Navigating the 2024 Gartner® Market Guide for DataOps Tools: What You Need to Know
Explore key findings from the 2024 Gartner® DataOps Market Guide. Enhance your data operations with these insights.
ETL for Beginners: Data Ingestion at Scale with S3 and Snowflake
This tutorial provides step-by-step instructions on how to set up a data ingestion pipeline to automatically ingest data from S3 into Snowflake, running in production.
Join Us at Airflow Summit 2024: Discover the Future of Apache Airflow with Astronomer
Join Astronomer at Airflow Summit 2024! Explore Astro, attend expert sessions, and network at our exclusive afterparty. See you there!
Customizing LLMs Through Astro
A Cost Sensitive Approach to Scalable Model Personalization Through Airflow
Data Products: It's not what you call them that matters. It’s what you do with them
Explore how data products are evolving, the challenges of modern data pipelines, and the future of unified orchestration and observability.
The AI Spring: How Demand for Production-Ready GenAI Projects is Continuing to Grow
Recent skepticism about Generative AI is healthy, for the short-term impacts are often exaggerated. But the long term impacts are most certainly profound.
A Musical Interlude: How Orchestras Inspire Modern Data Orchestration
Like a conductor, Airflow understands the flow of data, how a network of operations comes together to yield a data product.
The Need for Full-Stack Orchestration in the Age of the Data Product
The way we develop, orchestrate and observe data products needs to change. Learn how to get started by downloading our new guide.
Introducing Apache Airflow 2.10
The Airflow 2.10 release brings greater flexibility and expansion of some of the most widely used Airflow features.
A Step-by-Step Guide to Automating Your Astro Infrastructure with the Astro Terraform Provider
Learn to automate Astro onboarding with Terraform! This tutorial shows you how to create and manage workspaces.
Airflow and dbt: the next chapter
Introducing a unified approach to orchestrating dbt and Airflow with Astro
What’s New in the Astro Platform Release
We are excited to announce the new release of the Astro Platform, introducing exciting new features designed to enhance your data orchestration experience.
Understanding Airflow Trigger Rules: A Comprehensive Visual Guide
Discover the intricacies of Airflow trigger rules with visual examples and practical applications. Learn how to define and use various trigger rules to optimize your DAGs efficiently in Airflow. Essential reading for Airflow users working with version 2.9.2.
Astronomer Adopts DAG Factory to Democratize Writing Data Pipelines
We are thrilled to announce that Astronomer is officially taking over Adam Boscarino’s DAG Factory, an open source project that allows DAGs to be generated from YAML files.
Introducing Cohort 3 of the Astronomer Champions Program for Apache Airflow®!
We are thrilled to introduce Cohort 3 of the Astronomer Champions Program for Apache Airflow!
Exploring Airflow 2.9 Features with Astronomer's 29 Days of Airflow 2.9 Series
Explore Airflow 2.9's key features in Astronomer's 29 days of Airflow 2.9. Learn about dynamic scheduling, task management, and more.
Introducing the First Generative AI Cookbook for Data Orchestration
One of the biggest questions we get asked when discussing data orchestration for GenAI is how to get started. That is what our new GenAI Cookbook is designed to answer.
Announcing "The Data Flowcast" by Astronomer: Your Gateway to Mastering Airflow
Announcing The Data Flowcast: A podcast dedicated to all things Apache Airflow. Tune in for expert insights and trends.
SnowPatrol Series: Convert anomalies into actions, and how Grindr saved $600,000 in Snowflake costs
SnowPatrol helps Grindr save $600k in Snowflake costs. Discover how to use ML for anomaly detection and cost management.
Data Orchestration: The Dividing Line Between Generative AI Success and Failure
Learn how Apache Airflow® and Astronomer streamline data and model orchestration for Generative AI success. Explore practical use cases and a comprehensive guide.
Introducing Apache Airflow® 2.9
The Airflow 2.9 release brings significant enhancements to user-favorite features like data-aware scheduling, dynamic task mapping, and object storage.
Comprehensive Guide to Data Pipeline Testing with Airflow
An introduction to data pipeline testing strategies, best practices, and implementation techniques.
Introducing Cohort 2 of the Astronomer Champions Program for Apache Airflow®!
Our beta cohort of 10 is now joined by 23 hand-selected individuals who, we believe, truly embody what it means to champion the Apache Airflow® Project.
What’s new in the Astro Platform Release, Q1 2024
Welcome to the latest Astro Platform release — we’re thrilled to introduce enhancements aimed at bolstering governance at scale and across environments, fortifying the security of your data platform, and accelerating innovation.
Snowflake Anomaly Detection with SnowPatrol
Learn how SnowPatrol leverages machine learning and Airflow to detect anomalies in Snowflake usage, optimize costs, and improve data pipeline efficiency.
Improving Ask Astro: The Journey to Enhanced Retrieval Augmented Generation (RAG) with Cohere Rerank, Part 4
Our dedication to delivering top-tier products drove us to integrate Hybrid Search from Weaviate and the Cohere Rerank into our existing Ask Astro system.
Incident Management at Astronomer: 1 Year Later
One year ago, we rolled out a new incident management process. Read on to find the past, present, and future of incident management at Astronomer.
Reliable Data Orchestration for AI Applications
Discover how Dosu leverages Astronomer to streamline data orchestration for AI applications, ensuring reliable pipelines and boosting productivity. Learn how this partnership enhances AI development and supports open-source communities.
Tracking Innovation: How Astronomer Streamlined Release Notes with Towncrier
Discover how we created an efficient release note system with Towncrier. Minimized friction and improved developer workflows.
Maximizing Data Workflow Efficiency: The Advantages of Using Airflow with Azure Data Factory
ADF excels in creating quick and low-code data jobs with an intuitive UX. By layering in Airflow’s expressiveness orchestration capabilities on top of ADF workflows, developers get end-to-end visibility of their workflows without needing to migrate any jobs.
Welcome to our new New York City headquarters!
Astronomer has moved! At the start of this year, we relocated our headquarters to the heart of New York City at 50 West 23rd Street to support our growing business and customer base
How Astro runs billions of Airflow tasks around the world
A walkthrough of Astro's architecture and deployment model
Astro by Astronomer Delivered 438% ROI: Insights from a Forrester TEI Study
The value of Astro: a summary of Forrester’s Total Economic Impact™ Study
Standardizing your Astro projects with Cookiecutter and Cruft
A demonstration of how a platform team can develop a template Astro project for bootstrapping Astro projects for development teams. We demonstrate how to use Cookiecutter for developing a template project and Cruft for synchronizing generated projects with changes in the template project.
Revolutionizing Data Orchestration and MLOps with Apache Airflow® and Astronomer at Chiper
In this post we will cover how we had implemented a solid MLOps pipeline and data orchestration with the help of Airflow in multiple use cases.
Introducing the Astronomer Champions Program for Apache Airflow®
Today, we're thrilled to announce the launch of the Astronomer Champions Program for Apache Airflow®, a global initiative designed to recognize and empower outstanding data practitioners who are dedicated advocates of this powerful open-source orchestration tool.
Introducing Airflow 2.8
The latest minor Airflow release includes new features and improvements such as the Airflow ObjectStore, Listener hook for Datasets, enhanced logging capabilities, and more.
Deploy Rollbacks: Upgrade Airflow and Deploy DAGs with Confidence
The new Deploy Rollbacks feature enables users to revert any code deployed to Astro Deployments, including upgrades, to a known "good" state. This allows users to quickly recover from failing pipelines and avoid critical downtime.
Advanced XCom Configurations and Trigger Rules Tips and Tricks to Level-Up your Airflow DAGs
Master Airflow's Trigger Rules & XComs for flexible, resilient data pipelines. Learn how to handle complex scenarios and ensure flawless workflow execution.
Introducing the Astro Platform Release, Q4 2023
Unveiling Astro's latest features for streamlined connectivity, confident upgrades, and cost-efficient scaling. In this article, we’ll dive into these key features and explore how they can benefit your organization.
Enhanced Authentication Security to your Data Services on Azure with Astro
Experience advanced authentication with Apache Airflow®™ on Astro, the Azure Native ISV Service. Securely orchestrate data pipelines using Entra ID. Follow our step-by-step guides and leverage open-source contributions for a seamless deployment experience.
Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools
Build production-ready ML applications with Airflow's integrations for LLMs and AI.
Apache Airflow® TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs
Explore the TaskFlow API and traditional operators and find out how to combine them for dynamic, efficient DAGs.
Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider
See how the new Tecton Airflow Provider can make your feature pipeline orchestration within Apache Airflow® more efficient.
Ask Astro: Operationalizing Data Ingest for Retrieval Augmented Generation with LLMs, Part 3
Data ingestion for RAG with LLMs: Ask Astro Part 3 covers vector stores, schema design, and chunking.
Using Astronomer’s new Cosmos to deploy dbt pipelines onto Snowflake
The state of deploying pipelines with dbt has changed considerably in the last few months. Over the last few weeks, I was working with Astronomer to test out their new tool, Cosmos, to deploy dbt workflows onto Snowflake.
Databricks vs. Airflow From a Management Perspective, Part 2
Databricks vs Airflow from a production management perspective. Explore the differences in setup, monitoring, integrations, scalability & customization.
ML for Customer Analytics with Airflow, Snowpark, and Weaviate
An example project showing how to use Apache Airflow® to orchestrate a machine learning pipeline with the Snowpark provider and Snowpark ML.
Migrate Python Jobs to Airflow in 4 Simple Steps
Learn how easy it is to migrate Python scripts to Airflow DAGs, streamline orchestration, and leverage Airflow features to boost job efficiency.
3 Key Takeaways from Airflow Summit 2023
Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.
Ask Astro: An open source LLM Application with Apache Airflow®, Part 2
Build an LLM-powered chatbot with Airflow! Learn how to leverage domain-specific knowledge to create intelligent applications like "Ask Astro." Astro LLM meets Apache Airflow.
Day-2 Operations for LLM Applications with Apache Airflow®
Explore how Apache Airflow® solves day-2 operations challenges for LLM applications. Learn about scalable, reliable, and auditable workflows with Airflow.
Advanced Airflow CDC Implementation
Dive deeper into Airflow CDC implementation. Explore advanced use cases, best practices, and handle schema evolution & log-based sync effectively.
Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow®, Part 1
Choosing the right data orchestration tool for your needs can be tough. This blog post compares Databricks Workflows and Apache Airflow, two popular options.
Debugging Airflow Made Easy: 3 Key Steps to Debug your DAGs
Learn how to debug Airflow DAGs in 3 key steps. Eliminate common issues, set up a local development environment, and implement testing for seamless Airflow workflows.
Orchestrating Machine Learning Pipelines with Airflow
In the realm of machine learning, managing workflows efficiently is paramount. One tool that has emerged as a game-changer in this space is Apache Airflow®.
Change Data Capture (CDC) in Airflow: A Beginner's Guide
Understand the basics of Change Data Capture (CDC) in Airflow. Learn its importance, use cases, and core concepts for data pipeline success.
Introducing Airflow 2.7
The latest minor release includes several new features, such as automatic setup/teardown of tasks, built-in OpenLineage support, cluster activity view, fail-stop functionality, and more.
Leveraging Apache Airflow® and Kubernetes for Data Processing
Get insights on how to use Apache Airflow® and the Kubernetes Executor for data processing, along with proven best practices and tips for scaling your workloads.
Test Airflow Upgrades with the Astro CLI
The Local Upgrade Test command in the Astro CLI eliminates upgrade pains and ensures safe upgrades, allowing users to confidently identify and resolve compatibility issues, and DAG import errors.
Enhanced Astro Workspace Roles for more granular permissions
In this blog post, we will dive into the details of the Astro’s Role-Based Access Control (RBAC) and new Workspace Role updates, and explore improvements to popular use cases of Astro.
ETL in Airflow: A Comprehensive Guide to Efficient Data Pipelines
Optimize your data pipelines with Apache Airflow®. This guide covers tips for faster, more reliable, and easier-to-manage ETL workflows.
Run ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud
Use Apache Airflow® with CrateDB to run ETL processes, deploy with ease thanks to Astro and CrateDB Cloud.
Introducing Cosmos 1.0: the best way to run dbt Core in Airflow
The easiest way to orchestrate dbt Core using Apache Airflow®
6 Lessons Learned in Building Astronomer’s Developer Documentation
The Astronomer Approach to Clear and Effective Technical Documentation
Airflow Monitoring: Mastering SLAs, DAGs, & Observability
Learn how to effectively monitor Airflow DAGs, track SLAs, and maintain data pipeline health. Explore Airflow UI, notifications, and advanced observability tools.
Advantages of Hosted Airflow for Your ETL Workflows
Maximize ETL efficiency with hosted Apache Airflow® on Astro, not self-hosting open-source. Benefit from simplified infrastructure management, scalable elasticity, and dedicated support for your workloads.
Best Practices for Building an Airflow Service (Part 1)
This guide covers best practices for everything from choosing the right Airflow deployment model to configuring your DAGs for optimal performance.
Astronomer and Fivetran Partner to Release Production-Grade ELT Airflow Provider
The new provider will make it easier for organizations to use Airflow to automate and manage their Fivetran pipelines.
Astronomer and Snowflake: Unleash the Power of Snowpark Container Services and Apache Airflow®
Announcing the Astronomer and Snowflake partnership! Transform your data pipelines with Snowpark and Airflow.
Three ways to use Airflow with MotherDuck and DuckDB
Use Apache Airflow® with DuckDB and MotherDuck in three different ways. Access the DuckDB Python package directly, leverage the DuckDB Airflow provider, and use DuckDB with the Astro Python SDK.
The Astro Cloud IDE: from Python and SQL to nearly 1,000 Airflow operators
Use ~1,000 open-source Airflow operators and define your own custom operators in the Astro Cloud IDE with the newly released cell type functionality
The Top 7 Alternatives to MWAA
Beyond MWAA: Top 7 data orchestration tools for optimized workflows and increased productivity.
The Top 7 Alternatives to Google Cloud Composer
Picking the right tools for your data stack depends on your exact business and engineering needs, and the choice may seem daunting. Thankfully, there are several popular tools, each with thousands of users, all with a unique approach for managing data pipelines.
How we optimized the Registry for performance across millions of page views
Migrating the Astronomer Registry’s backend from Airtable to Postgres and a Golang REST API
Scale Airflow with confidence using Astro’s new Alerting capabilities
This new functionality in Astro lets you easily implement DAG-level or task-level alerts to be sent to Slack or PagerDuty.
Introducing Airflow 2.6
Apache Airflow® 2.6 contains over 500 commits from over 130 contributors, adding up to 35 new features, 50 general improvements, and 27 bug fixes.
Kubernetes Executor Support in Astro: A New Era of Scalability and Resource Management
Announcing Kubernetes Executor support in Astro. You can now take advantage of the power of Kubernetes to manage resources & scale your Airflow workloads.
Astro CI/CD Enforcement for Code Changes
Ensure that all code changes are deployed within your CI/CD processes, increase code quality, and enforce automated testing.
OpenLineage Is on the Rise in 2023
Learn why OpenLineage is catching on and see what lies ahead for this open-source standard.
Inside Authorized Workspaces, A New Feature in Astro
Authorized Workspaces, a new feature in Astro, lets customers isolate teams or projects to specific clusters in their data planes.
Introducing Support for the Kubernetes Executor on Astro, Now in Private Preview
The Kubernetes Executor offers Astro customers task isolation, efficient resourcing, and simplicity.
Improving a Data Quality Process by Adding Great Expectations
Simplify data quality checks in Airflow with Great Expectations. Learn how to integrate, set up, and leverage its powerful features for reliable pipelines.
Introducing Astro’s New Workspace Homepage
The updated Astro homepage brings together the key pieces of information a user needs to start their day.
The Airflow Year in Review 2022
Find out how Airflow has been optimized in 2022. Learn about major updates, including data-driven scheduling, dynamic task mapping, and UI enhancements.
Win a Scholarship for CoRise’s “Effective Data Orchestration with Airflow” Course
Learn how to win a full scholarship to CoRise’s new Airflow and data orchestration course.
The New, Faster Way to Deploy Airflow DAGs to Astro
The new DAG-only deploy feature in the Astro CLI makes deploys to Astro significantly faster and allows for more flexibility in CI/CD workflows.
How an Improved DAG-Testing Command in the Astro CLI Made Its Way into Airflow
Learn how improved DAG-testing commands in the Astro and Airflow CLIs make DAG authoring easier and help DAGs run more reliably.
5 Ways to View and Manage DAGs in Airflow
Find out what the most popular and useful DAG views in the Airflow UI are. Learn about the Airflow Graph View, Grid View, Calendar View, and Browse Tab.
Introducing the Astro Cloud IDE
Discover the Astro Cloud IDE, a notebook-inspired tool for writing data pipelines. See how to define tasks and connections without knowing Apache Airflow®.
What’s New in Apache Airflow® 2.5
Check out what’s new in Apache Airflow® 2.5. Learn more about improvements to Airflow’s dynamic task mapping and data-dependent scheduling features.
A Short History of DAG Writing
Learn about Airflow & its updated features. Get to know how users can benefit from Taskflow API, Custom XCom Backends, Astro SDK, and the Astro Cloud IDE.
Best Practices for Secure Network Connectivity and Authentication in Astro
Learn how to securely hook up data sources and implement strong authentication in Astro — a modern data orchestration solution powered by Apache Airflow®.
3 Ways to Extract Data Lineage with Airflow
Learn how to extract data lineage events from your Airflow pipelines using OpenLineage. Plus, see how these three methods work with the Astro platform.
How Astro’s Data Graph Helps Data Engineers Run and Fix Their Pipelines
Discover how Astro can help you understand, communicate, and solve pipeline problems. Learn about the key pipeline observability feature – the Data Graph.
OpenLineage: Where It Came from and What Comes Next
Hear from Julien Le Dem, Chief Architect at Astronomer, about the creation of OpenLineage and how it’s evolving into a standard for data lineage.
How to Keep Data Quality in Check with Airflow
Learn more about Airflow-driven data quality checks, their benefits, and design. Find out how data quality issues are detected and solved at Astronomer.
What’s New in Astro Python SDK 1.1: Data-Driven Scheduling, Dynamic Tasks, and Redshift Support
Learn what the new upgraded Astro Python SDK 1.1 offers to Airflow users. Find out more about data-driven scheduling, dynamic tasks, and Redshift support.
Orchestration, or How to Become a Data-Driven Company
Learn how Astro, the modern Airflow-powered data orchestration platform, helped Astronomer build a fully coordinated data ecosystem.
Micropipelines: A Microservice Approach for DAG Authoring in Apache Airflow®
Learn how to tune the data system and have critical data products ready on time with micropipelines. See how to make DAG authoring easier with Airflow 2.4.
Expanding Data Access and Exchange Inside a Company
Hear from Steven Hillion and Taylor Merrick about how our data scientists combine tooling and process to encourage company-wide data product development.
Airflow 2.4 and Data-Driven Scheduling: How a New Feature Is Saving Time at Astronomer
Learn how Astronomer is using the new data-driven scheduling feature in Airflow 2.4. See how it benefits DAG authors and helps solve timing problems.
Astro CLI: The Easiest Way to Install Apache Airflow®
Learn all about the Astro CLI — the free, open source tool that makes it easy to install, run, and test Apache Airflow® from your command line.
Apache Airflow® 2.4 — Everything You Need to Know
Discover the newly released Apache Airflow® 2.4. Find out how its new data-driven scheduling logic enables faster and easier delivery of data.
Podcast Spotlight: What Observability Brings to Data Orchestration
Hear from Astronomer’s Senior Vice President of R&D about how data orchestration and observability improve the quality and reliability of dataflows.
Announcing Astro’s HIPAA and PCI-DSS Compliance
Two more reasons that users who need high levels of data security and protection can count on Astro.
How We Track the Growth of Apache Airflow®
Find out how we use data to keep track of what’s happening in the Airflow project.
Reimagining Airflow for Data Engineers and Data Scientists with the Astro Python SDK
Discover the Astro Python SDK—an open-source framework for writing Airflow data pipelines.
Astro Is Now Available on All Major Cloud Providers
Astro is now available on AWS, Microsoft Azure, and Google Cloud in 47 regions across six continents. Learn more on our blog.
Everything You Should Know About Airflow 2.3’s New Grid View
Learn about Airflow 2.3’s new grid view. Find out how to easily visualize complex representations in Airflow’s UI with this long-awaited intuitive feature.
The Astronomer Providers Package — A Better Option for Long-Running Tasks
Learn about Astronomer Providers — a collection of open-source operators, hooks, and sensors that allow you to schedule long-running tasks asynchronously.
Ventana Research Names Astronomer as a Finalist for Its Digital Innovation Awards
Astronomer is a finalist for the 15th Annual Ventana Research Digital Innovation Awards, recognized for innovative technologies in their markets.
Introducing Astro, the Fully Managed Data Orchestration Platform, Powered by Airflow
Discover Astro – the data orchestration platform powered by Airflow. Find out how to build, run, and observe data pipelines efficiently and with context.
Introducing New Astro CLI Commands to Make DAG Testing Easier
Learn how the new Astro CLI commands provide a great DAG development experience for Airflow users. Get to know the dev parse and dev pytest commands.
To Build or to Buy? DIY Orchestration with Airflow vs. A Fully Managed Service
Hear some reasons organizations consider building their own Apache Airflow® infrastructures, and how a fully managed service makes you more competitive.
Introducing Astronomer Providers
Find out more about Astronomer Providers, a set of Airflow 2-licensed providers with async functionality, created and maintained by Astronomer experts.
Apache Airflow® 2.3 — Everything You Need to Know
Learn what’s new in the latest release of Apache Airflow® 2.3 and how it can improve data orchestration.
Apache Airflow® for Data Scientists
Learn all about the role and challenges of a data scientist and find out how Apache Airflow® can help with your workflows.
10 Best Practices for Modern Data Orchestration with Airflow
Learn best practices for standing up, scaling, and growing Apache Airflow® to support modern data orchestration.
Airflow and dbt, Hand in Hand
Learn how to use Airflow and dbt together to advance data orchestration and data transformation projects and facilitate collaboration across data teams.
What Is Data Lineage and Why Does It Matter?
Discover the power of data lineage and its role in improving data observability and quality. Find out how to take data orchestration to the next level.
Astronomer Acquires Datakin, the Data Lineage Tool
Learn how Astronomer acquired Datakin, the real-time, operational data lineage tool from the founders of the OpenLineage and Marquez open-source projects.
Letter from the CEO: Our Story So Far
Hear how Joe Otto reflects on Astronomer’s history, and looks to a future powered by the combination of orchestration, lineage, and observability.
TechCrunch on Astronomer’s Big News
The site covered our recent acquisition of Datakin and our Series C round.
Apache Airflow® for Data Leaders — How to Empower Data Teams
Learn the common challenges data and analytics leaders face and how they use Apache Airflow® and Astronomer to empower themselves and their data teams.
Airflow Summit 2022 — Join the Airflow Event of the Year!
The biggest community-driven event around Apache Airflow® returns May 23–27, 2022.
Apache Airflow® at Astronomer—Taking Data Orchestration to the Next Level
Learn how Astronomer became the top data orchestration platform based on Apache Airflow®. See how to apply Airflow in your ETL and analytics use cases.
Airflow Best Practices
Master Apache Airflow® with these 10 best practices. Learn how to optimize your data pipelines, improve efficiency, and avoid common pitfalls.
Top Data Management Trends for 2022
Learn about emerging trends that are revolutionizing the world of data from the leading Apache Airflow® experts. See how to efficiently manage data in 2022.
Adding Data Quality to DAGs ft. Great Expectations
Find out how to use Great Expectations in an Airflow Directed Acyclic Graph to successfully perform, prioritize, and schedule data quality checks.
Astronomer and Uturn Partner to Drive Innovation and Better Business Outcomes
Learn more about Astronomer’s new partnership with Uturn Data Solutions – the leading experts in enterprise cloud enablement and application modernization.
Apache Airflow® for Data Engineers—How to Leverage Data Orchestration
Gain insight into the role and responsibilities of a data engineer. See 7 examples of how Apache Airflow® can make data engineering less challenging.
How to orchestrate Azure Data Explorer queries with Airflow
Learn how to orchestrate Azure Data Explorer queries with Airflow.
How to Select the Best ETL Tool to Integrate With Airflow? Our 3 Picks
Learn from our experts about how to select the best ETL tool, and why it pays to integrate Fivetran, Airbyte, and Azure Data Factory with Apache Airflow®.
Every Company Nowadays Becomes a Data Company—Interview with Bolke de Bruin
Hear from Bolke de Bruin – VP of Enterprise Data Services at Astronomer – about how Apache Airflow® helps modern companies manage data effectively.
Machine Learning Pipeline Orchestration
Learn about machine learning orchestration, machine learning pipelines, and their components. See why Apache Airflow® is the top ML data orchestration tool.
How to Build a Modern Data Stack
Breaking down what a modern data stack means in practice. We discuss four core components, five reasons to set it up, and how to orchestrate it.
Apache Airflow® vs. Apache Beam: A Comparative Guide
Explore the differences and similarities between Apache Beam and Airflow. Understand their capabilities, programming models, and ideal use cases to make the right choice for your data management needs.
Democratizing the Data Stack—Airflow for Business Workflows
See why modern orchestrators and reverse ETL tools are the future for data-driven business. Learn how Apache Airflow® takes SQL to the next level.
Machine Learning Pipeline: Everything You Need to Know
Discover what a machine learning pipeline is and the process behind creating one with Apache Airflow®. Learn what you need to know about ML pipelines.
What is Reverse ETL and How Can It Improve Data Flow?
Understand the difference between ETL and reverse ETL. Learn to use the Census reverse ETL platform and Airflow together to leverage data orchestration.
Airflow at BBC—Data Orchestration Solution in Media
Hear what the BBC's Data Engineer says about the popularity of orchestration tools in the media industry. Find out why the BBC went for Apache Airflow®.
Everything You Need to Know About Apache Airflow® 2.2.0
See what’s new in Apache Airflow® 2.2. Learn about big improvements, bug fixes, and internal changes, and the benefits they bring to Apache Airflow® users.
Big Data Architecture: Core Components, Use Cases, and Limitations
Learn more about the concept of Big Data Architecture and its 5 core components. See how various companies benefit from implementing Big Data Architecture.
The Future of Banking: How Can Apache Airflow® Help?
Learn how to make the most of data in banking. See which banks use Airflow, and how the top orchestration tool helps them overcome major challenges.
Apache NiFi vs. Apache Airflow®
Get to know the major benefits and limitations of Apache NiFi and Apache Airflow, and see which of the two popular ETL tools is better for data management.
Airflow at Wise: Data Orchestrator in Machine Learning
Alexandra Abbas—a Machine Learning Engineer at Wise—explains what makes Airflow an ideal tool for data orchestration in the fintech industry.
How to Build an ETL Process?
Learn what an ETL process is and how to build it. Find out how Apache Airflow® can help you create, scale, and manage ETL pipelines more effectively.
Data Silos: What Are They and How to Fix Them?
Find out what causes data silos and how they hurt your business. Learn how Airflow and data orchestration can solve the data silo problem in your company.
Airflow at Societe Generale: Data Orchestration Solution in Banking
Hear from the Product Owner at Societe Generale about the benefits of implementing Airflow. Find out how data orchestration solutions are used in banking.
Data Pipeline: Components, Types, and Best Practices
Learn the basics of data pipeline building. Get to know data pipeline components, types, and best practices. See how Airflow can simplify the process.
Generating Airflow DAGs from dbt Models (Part 3)
Streamline DAG generation! Explore a utility package to create Airflow DAGs from dbt models. Get sample configurations & customizable code.
What Is Data Orchestration and Why Is It Essential for Business
Data orchestration is the process of collecting siloed data from multiple locations and systemizing, unifying, and activating it for data analysis.
Airflow Summit 2021 Highlights
Learn what the Airflow community got up to in 2021, in this recap of the biggest international Airflow event. Get ready for the next Airflow Summit!
How Data Pipelines Drive Improved Sales in E-commerce
Hear from Viraj Parekh–Field CTO at Astronomer–about how data pipelines help increase online sales. See why Apache Airflow® works for e-commerce companies.
Everything You Need to Know About the Airflow Summit 2021
Learn why it’s worth attending the biggest Airflow conference for developers and data professionals. Check what’s on the agenda and register for free.
Validate Your Apache Airflow® Skills With the Astronomer Certification
Find out how to get started with Apache Airflow® and enhance your knowledge. Learn essentials from Astronomer’s experts and become Apache Airflow®-certified!
The New KubernetesExecutor
Learn more about the KubernetesExecutor and its upgrade to version 2.0. See new features redesigned with Airflow admins and data engineers in mind.
How to orchestrate Talend jobs with Airflow
Learn how to orchestrate Talend jobs with Airflow so you can use both tools without rewriting your pipelines.
Announcing the Astronomer Registry
Learn about the discovery-and-distribution hub for Airflow integrations. See how to bridge the gap between the Airflow community and the data ecosystem.
Airflow 2.0 TaskFlow API and Its Features
Learn more about the TaskFlow API and read about its features. Get to know how TaskFlow API in Airflow 2.0 enables a better DAG authoring experience.
Airflow Secrets Management: Best Practices for Airflow 2.0
Have a look at Astronomer’s ultimate guide on Airflow Secrets, and learn best practices for managing Secrets with various backends in Apache Airflow® 2.0.
Near-Real-Time CDC with Airflow and GCP
Learn how to implement near-real-time Change Data Capture (CDC) in Airflow using a scheduled GCP CloudSQL export approach for data pipelines.
Deploying Airflow DAGs in Production with dbt (Part 2)
Take your Airflow DAGs live! Learn how to deploy them in production using dbt manifest.json, and integrate dbt into your ETL/ELT workflows.
Building a Scalable Analytics Architecture With Airflow and dbt (Part 1)
Kickstart your analytics architecture with Airflow and dbt. Learn DAG authoring, configurations, and code snippets for a seamless setup.
The Airflow 2.0 Scheduler
Explore the features of the updated Apache Airflow® 2.0 Scheduler. Learn how the Airflow Scheduler enables quick and seamless initiation of tasks.
A Great Expectations Provider for Apache Airflow®
Find out why Great Expectations and Apache Airflow® are a great match. Learn how to leverage native Great Expectations functionality directly from DAGs.
Introducing Airflow 2.0
Get to know the highlights of Apache Airflow® 2.0 and see hundreds of new features it includes. Have a look at how Airflow 2.0 compares to Airflow 1.10.
Introducing KEDA for Airflow
Explore the possibilities of the Kubernetes Event-Driven Autoscaler. See how KEDA helps users improve the efficiency of their Apache Airflow® deployments.
Profiling the Airflow Scheduler With Flame Graphs
Get tips on improving the performance and reliability of the Airflow Scheduler. Find out how to benchmark and profile it using py-spy and Flame Graphs.
Why Airflow?
Discover Apache Airflow® and explore its workflow-management capabilities. See which global companies use Airflow to solve data engineering challenges.
The Next Generation of Astronomer Cloud
Learn how Astronomer Cloud supports the latest version of Apache Airflow®. See the features included in the newly released, next-generation data platform.
Astronomer v0.10
Learn more about Astronomer v0.10 and its key updated functionalities. See how the new Astronomer Platform supports the latest version of Apache Airflow®.
7 Common Errors to Check When Debugging Airflow DAGs
Get to know best practices for debugging Apache Airflow® DAGs. Check out the list of common Airflow deployment errors, and see how to find and remove them.
Airflow Design Principles: Multi-tenant vs. Monolithic Architecture
Find out how to design an Airflow infrastructure, and whether it makes more sense to power your DAGs with one monolithic Airflow instance, or many.
Astronomer v0.8.0 Release Notes
Discover the newly launched Astronomer v0.8.0 and the features it includes. Find out what’s been fixed, improved, and added to the Astronomer Platform.
Astronomer on Astronomer: Loading Thousands of Files Into Redshift With Apache Airflow®
Get to know why Astronomer switched to Apache Airflow®. Learn how we optimized Airflow to fit our initial needs and what features we're planning to build.
Astronomer v0.7.0 Release Notes
Release notes covering the features released with v0.7.0 of the Astronomer platform.
Astronomer v0.6.0 Release
Release notes for v0.6.0 of the Astronomer platform.
Astronomer v0.5.0 Release
Release notes from our recent platform update to v0.5.0.
Astronomer v0.4.1 Release
Check out the highlights of the Astronomer v0.4.1 release. See the full summary of upgrades and learn more about the Astronomer Platform's new features
How the Apache Airflow® Project Will Change
Explore the future of Apache Airflow® at Astronomer. Take a look at the official Airflow roadmap and see what improvements and developments to expect.
Astronomer v0.3.2 Release
Learn more about Astronomer v0.3.2 and its new, updated functionalities. Find out what’s been changed, fixed, and added to the Astronomer Platform.
Announcing Astronomer v0.3
Find out more about Astronomer v0.3 and its great benefits. Get to know what features are included in the newly released next-generation data platform.
Announcing the Astronomer Platform, a Managed Service for Apache Airflow®
Learn about the benefits of the Astronomer’s Managed Service for Apache Airflow®. See why our experts decided to design and build the Astronomer Platform.
Announcing Astronomer SpaceCamp
Discover Astronomer SpaceCamp and see how it gets data teams up and running with Airflow in no time. See the benefits of different SpaceCamp versions.
Announcing The Airflow Podcast
Learn about Astronomer's podcast focused on the future potential of Apache Airflow®, as seen by top players in the data engineering space.
An Airflow Story: Cleaning and Visualizing Our Github Data
Check out instructions on importing and moving Github data using Apache Airflow®. See how to deal with Github DAG writing, visualization, and dashboarding.
Improving Government Services With Apache Airflow®: a Q&A With San Diego’s Chief Data Officer
Hear from Maksim Pecherskiy–Chief Data Officer of the City of San Diego–about how Apache Airflow® helps operationalize data in the public sector.
From Behavioral Analytics to Data Science With Astronomer
Find out how to scale behavioral analytics with Apache Airflow®. See why the Astronomer Platform is an ideal solution for data scientists and analysts.
Using Apache Airflow® to Create Data Infrastructure in the Public Sector
Learn why ARGO chose Apache Airflow® to build and maintain data infrastructure. Find out how they transform basic public services with Airflow.
Data Format Types: CSV, JSON, & XML Demystified
Explore common data format types (CSV, JSON, XML) & understand their pros, cons, & ideal use cases. Learn how to choose the right format for your data needs.
Why Every Data Scientist Needs a Data Engineer
Get to know the roles and responsibilities of data scientists and data engineers. Learn why data engineering and data science go hand in hand.
What Exactly Is a DAG?
Learn what a DAG is and how it's used in data pipelines. Explore benefits, real-world examples, and FAQs in this comprehensive guide.
Data Engineering Platform Astronomer Closes $3.5M Financing
Learn how Astronomer, a data engineering platform, closed $3.5M in new financing, led by Wireframe Ventures in San Francisco and CincyTech in Cincinnati.
Normalizing Data for Warehouse Centralization
Knowing the options for storing data will help you make the right decisions for your company when you’re ready to take this step.
Apache Airflow® and the Future of Data Engineering: A Q&A with Maxime Beauchemin
Hear from Maxime Beauchemin – a data engineer at Airbnb and creator of their data pipeline framework, Airflow – about the future of data engineering.
Our Open Source Philosophy
Get to know how the open-source approach helps drive growth and innovation. Learn why it’s worth investing in open-source components like Apache Airflow®.
Why Is My Data Playing Hard to Get?
Learn more about the different types and properties of hard-to-reach data with great potential. Find out how to access, organize, and store it effectively.
Airflow at Astronomer
Learn why Astronomer needed a unified scheduling system to extract and monitor all types of data pipelines. Find out why Apache Airflow® was our answer.
Press Release: Astronomer Announces Seed Financing
Press Release: Astronomer Closes $1.9M in Seed Financing
Our Unique Path to Raising $2M Seed in the Midwest
Learn about Astronomer’s 2-year journey to raising $2 million seed, including the ups and downs we went through to take our company to the next level.
Lessons Learned Writing Data Pipelines
See how to simplify the data pipeline writing process with the right tools. Learn what Astronomer experts do to make data pipelines less challenging.
Why We Built Our Data Platform on AWS, and Why We Rebuilt It With Open Source
Find out how Astronomer leveraged AWS services with open-source alternatives. See why Airflow and Apache Mesos help build more and better integrations.
An Almost Acquisition Story
Coming out of AngelPad’s 2015 Demo Day, we found ourselves vacillating between an acquisition and Series A, though we were arguably too early for either.
Announcing Astronomer v0.9
Release notes for v0.9 of the Astronomer Platform
A Logo Story
Astronomer's Head of Design, Chris Hendrixson, explains how he created the design aesthetic to encompass data, futurism, and a little bit of fun.
Setting Up Your Redshift Cluster
Redshift is popular but you still need to know what you''re doing when spinning up your first cluster. In this tutorial, we walk you through the process.
When Should You Start to Warehouse Your Data?
Learn how to leverage your business data with data warehousing. Discover the best time to create a data warehouse, and see which warehousing tools to use.
Why We Drove to NY and Back Over the Past 48 Hours for a 15-Minute Meeting
Find out why it was worth driving from Cincinnati to New York for a fifteen-minute meeting with the number one ranked startup accelerator in the world.
How We Built Cross-Region Disaster Recovery for Astro on Google Cloud
How we built it Cross-region disaster recovery on GCP: the architecture, the decisions specific to Google Cloud, and what teams need to plan before starting.
Cross-Region Disaster Recovery on Astro Is Now Generally Available: Here's How We Built It
Disaster Recovery on Astro is Generally Available. A look at the architecture decisions, trade-offs, and engineering lessons behind cross-region Disaster Recovery on Astro - from database replication with Aurora Global Clusters to warm standby compute and one-click failover
The EduAgent: The AI Agent Behind Astronomer Academy's Support
How the Astronomer Education team uses AI to help respond to user tickets and answer questions about Academy usage.
The AI Agent Behind Astro Runtime's Reliability
How Astronomer built an AI agent that reads every Apache Airflow commit daily to catch breaking changes before they reach customers.
Building Kepler
How Astronomer built Kepler, an internal Slackbot and CLI tool that answers ad hoc data questions using LLMs, hybrid search, and code context.
What Happens When a PM Runs a Coding Agent (almost) Full-Time
How a Principal Product Manager used AI coding agents like Claude Code during onboarding to automate workflows, track customer conversations, and ship features in his first 90 days at Astronomer.
Incident Management at Astronomer: 1 Year Later
One year ago, we rolled out a new incident management process. Read on to find the past, present, and future of incident management at Astronomer.
Tracking Innovation: How Astronomer Streamlined Release Notes with Towncrier
Discover how we created an efficient release note system with Towncrier. Minimized friction and improved developer workflows.
How Astro runs billions of Airflow tasks around the world
A walkthrough of Astro's architecture and deployment model
