Astronomer Blog
The latest insights from our team of Apache Airflow® experts.
Supercharge dbt Orchestration with Astronomer Cosmos and Apache Airflow®
Cosmos 1.11.0a1 introduces alpha support for dbt Fusion—the next-generation dbt engine that unlocks lightning-fast parsing, state-aware orchestration, and real-time validation, all orchestrated natively in Airflow.
In an AI world, it’s the workflow that allows you to build your moat
While LM Arena leaderboards grab headlines, an infrastructure battle is happening in the orchestration systems that transform AI from lab experiments into production solutions that solve real-world problems.
Introducing Data Quality in Astro Observe: The Orchestration-First Approach to Reliable Data
Astronomer is excited to announce enhanced data quality monitoring in Astro Observe, now available in private preview.
Introducing Cohort 5 2025 of the Astronomer Champions Program for Apache Airflow®!
Meet Cohort 5 of the Astronomer Champions Program for Apache Airflow in 2025. Data leaders from top global organizations are driving data engineering's future.
Secure, Flexible Data Orchestration: Meet Remote Execution on Astro
With Remote Execution on Astro, enterprises no longer have to choose; you can now run workloads exactly wherever they need to be, while still benefiting from centralized orchestration and observability in Astro.
Why Enterprise AI Struggles: The Context Gap, Data Gravity, and What Comes Next
Discover why context is vital for effective AI implementation and how shifting from centralized data systems to context-rich approaches enhances decision-making.
Native support for Source Node Rendering in Cosmos
Cosmos now natively supports source node rendering! Visualize your dbt DAG more effectively with this enhanced feature. Learn how it simplifies data lineage.
The Operating System for Enterprise AI
In the AI era, orchestration platforms become the lynchpin.
From Regulation to Resilience: How Astronomer Powers DORA-Ready Data Operations
The Digital Operational Resilience Act (DORA) addresses the critical need for the financial sector to effectively manage digital operational resilience in the face of rising cyber threats and Information and Communication Technology (ICT) disruptions.
Workflows then agents: the practical approach to enterprise AI
Introducing the AI SDK for Apache Airflow™
Enhanced Alerting at Astronomer: Streamlined Airflow Monitoring at Scale
Astro Enhanced Alerting provides a unified monitoring experience through pattern-based alert rules that teams can apply seamlessly across hundreds of DAGs simultaneously.
Astronomer and IBM Collaborate to Transform Enterprise Data Orchestration
Astronomer and IBM join forces to revolutionize enterprise data orchestration! Discover how this exciting collaboration will transform data management.
Introducing new flexible billing options for Astro
You can now upgrade to our Team plan while continuing to only pay for what you use, billed monthly to a credit card or cloud marketplace.
Apache Airflow® 3 Development Update
Airflow 3.0 is the biggest release in Airflow’s history, the result of a massive effort from the global Airflow community.
State of Airflow 2025: Unleashing the Future of Data Orchestration
Key Findings from the State of Airflow Report 2025
Airflow in Action: Customizing LLMs at Laurel. Speeds Rollout and Saves $500k / Year in Inference Through GenAI Ops
How Laurel uses Airflow to scale GenAI, automate timekeeping, and cut LLM costs by $500K per year. Learn how DAGs power model retraining and inference.
Astro Observe is Now Generally Available
The new standard for pipeline reliability and data product observability.
FreightWaves scales Data Engineering with Astronomer and Apache Airflow
FreightWaves transitioned from a scattered and inefficient setup to Astronomer’s Astro, powered by Apache Airflow, to transform their data engineering practices.
Why Orchestration and DataOps Will Redefine the Modern Data Stack
DataOps is the future of the modern data stack. Learn how orchestration-first platforms like Astro are unifying the stack & delivering value.
Celebrating Innovators in Data Orchestration: the Astronomer Data Excellence Awards
Discover the winners of the Astronomer Data Excellence Awards! Celebrating groundbreaking achievements in data orchestration with Astro and Airflow.
Modernizing legacy ETL workloads with Airflow
How to Migrate Legacy ETL Workloads from Informatica to Airflow DAGs Using DAG Factory
Airflow in Action: 33% Faster Deploys with 90% Higher Data Quality. Astronomer at Autodesk
Discover Autodesk’s secure UAT environment for PII testing, achieving faster deployments and reduced technical debt, powered by Airflow and Astronomer.
TrackFly Optimizes Data Workflows with Astronomer and Apache Airflow
TrackFly turned to Astronomer’s managed Airflow solution, Astro, to streamline their ETL (Extract, Transform, Load) workflows and focus on core business goals.
Data Observability 101: An Introduction to the Most Critical Features of Modern Data Observability
Explore core data observability features: analytics, monitoring, and alerts. Gain insights into pipeline health, prevent failures, and ensure data reliability.
ELT for Beginners: Extract from S3, Load to Databricks and Run Transformations
Learn how to build an ELT pipeline extracting data from S3, loading it into Databricks, and transforming it with notebooks using Apache Airflow.
Customer Story: Autodesk’s Data Engineering Transformation with Astronomer and Apache Airflow
Autodesk scales with Astro! 🚀 Learn how they migrated from Oozie to Airflow, boosting efficiency, scalability, and data-driven decision making.
It Pays to be Picky: Leveraging Key Insights with Airflow and Astro
Here are four key observability insights to consider focusing on when setting up an observability solution: data freshness, on-time delivery, data dependencies tracking, and data quality.
Update: Astro Observe is Now In Public Preview
Today, we’re excited to announce Astro Observe, which brings new and more robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC).
Proactive Airflow Monitoring: How to Prevent Infrastructure Issues Before They Happen
Prevent Airflow downtime with proactive deployment health alerts. Automate monitoring & get actionable insights for reliable data pipelines.
Unlocking the Power of Scalable Machine Learning with Anyscale and Astronomer
Anyscale and Astronomer join forces to solve the challenges of scaling machine learning and AI.
Introducing Cohort 4 of the Astronomer Champions Program for Apache Airflow!
This final cohort for the 2024 calendar year brings together data leaders from Fortune 500 companies around the globe, all possessing extensive Airflow expertise.
Announcing Astronomers Vulnerability Disclosure Program
Astronomer launches Vulnerability Disclosure Program, a way for us to better engage with the global community in receiving, recognizing, and rewarding findings from the collective security community.
Best Practices and Solutions for Multi-Tenant Airflow
How Isolated Environments Enhance Execution, Security, and Productivit
Introducing Cosmos 1.6: The Best Way to Run dbt-core in Airflow
We're happy to announce that the newest version of Astronomer’s dbt-core integration, Cosmos 1.6.0, is now available, featuring a list of enhancements and some great new additions to serve the community.
Astro and Terraform: Empowering Infrastructure as Code for Modern Data Orchestration
Announcing the Astro Terraform Provider! Use Terraform to manage and automate your Astro deployments.
Navigating the 2024 Gartner® Market Guide for DataOps Tools: What You Need to Know
Explore key findings from the 2024 Gartner® DataOps Market Guide. Enhance your data operations with these insights.
ETL for Beginners: Data Ingestion at Scale with S3 and Snowflake
This tutorial provides step-by-step instructions on how to set up a data ingestion pipeline to automatically ingest data from S3 into Snowflake, running in production.
Join Us at Airflow Summit 2024: Discover the Future of Apache Airflow with Astronomer
Join Astronomer at Airflow Summit 2024! Explore Astro, attend expert sessions, and network at our exclusive afterparty. See you there!
Customizing LLMs Through Astro
A Cost Sensitive Approach to Scalable Model Personalization Through Airflow
Data Products: It's not what you call them that matters. It’s what you do with them
Explore how data products are evolving, the challenges of modern data pipelines, and the future of unified orchestration and observability.
The AI Spring: How Demand for Production-Ready GenAI Projects is Continuing to Grow
Recent skepticism about Generative AI is healthy, for the short-term impacts are often exaggerated. But the long term impacts are most certainly profound.
A Musical Interlude: How Orchestras Inspire Modern Data Orchestration
Like a conductor, Airflow understands the flow of data, how a network of operations comes together to yield a data product.
The Need for Full-Stack Orchestration in the Age of the Data Product
The way we develop, orchestrate and observe data products needs to change. Learn how to get started by downloading our new guide.
Introducing Apache Airflow 2.10
The Airflow 2.10 release brings greater flexibility and expansion of some of the most widely used Airflow features.
A Step-by-Step Guide to Automating Your Astro Infrastructure with the Astro Terraform Provider
Learn to automate Astro onboarding with Terraform! This tutorial shows you how to create and manage workspaces.
Airflow and dbt: the next chapter
Introducing a unified approach to orchestrating dbt and Airflow with Astro
What’s New in the Astro Platform Release
We are excited to announce the new release of the Astro Platform, introducing exciting new features designed to enhance your data orchestration experience.
Understanding Airflow Trigger Rules: A Comprehensive Visual Guide
Discover the intricacies of Airflow trigger rules with visual examples and practical applications. Learn how to define and use various trigger rules to optimize your DAGs efficiently in Airflow. Essential reading for Airflow users working with version 2.9.2.
Astronomer Adopts DAG Factory to Democratize Writing Data Pipelines
We are thrilled to announce that Astronomer is officially taking over Adam Boscarino’s DAG Factory, an open source project that allows DAGs to be generated from YAML files.
Introducing Cohort 3 of the Astronomer Champions Program for Apache Airflow®!
We are thrilled to introduce Cohort 3 of the Astronomer Champions Program for Apache Airflow!
Exploring Airflow 2.9 Features with Astronomer's 29 Days of Airflow 2.9 Series
Explore Airflow 2.9's key features in Astronomer's 29 days of Airflow 2.9. Learn about dynamic scheduling, task management, and more.
Introducing the First Generative AI Cookbook for Data Orchestration
One of the biggest questions we get asked when discussing data orchestration for GenAI is how to get started. That is what our new GenAI Cookbook is designed to answer.
Announcing "The Data Flowcast" by Astronomer: Your Gateway to Mastering Airflow
Announcing The Data Flowcast: A podcast dedicated to all things Apache Airflow. Tune in for expert insights and trends.
SnowPatrol Series: Convert anomalies into actions, and how Grindr saved $600,000 in Snowflake costs
SnowPatrol helps Grindr save $600k in Snowflake costs. Discover how to use ML for anomaly detection and cost management.
Data Orchestration: The Dividing Line Between Generative AI Success and Failure
Learn how Apache Airflow® and Astronomer streamline data and model orchestration for Generative AI success. Explore practical use cases and a comprehensive guide.
Introducing Apache Airflow® 2.9
The Airflow 2.9 release brings significant enhancements to user-favorite features like data-aware scheduling, dynamic task mapping, and object storage.
Comprehensive Guide to Data Pipeline Testing with Airflow
An introduction to data pipeline testing strategies, best practices, and implementation techniques.
Introducing Cohort 2 of the Astronomer Champions Program for Apache Airflow®!
Our beta cohort of 10 is now joined by 23 hand-selected individuals who, we believe, truly embody what it means to champion the Apache Airflow® Project.
What’s new in the Astro Platform Release, Q1 2024
Welcome to the latest Astro Platform release — we’re thrilled to introduce enhancements aimed at bolstering governance at scale and across environments, fortifying the security of your data platform, and accelerating innovation.
Snowflake Anomaly Detection with SnowPatrol
Learn how SnowPatrol leverages machine learning and Airflow to detect anomalies in Snowflake usage, optimize costs, and improve data pipeline efficiency.
Improving Ask Astro: The Journey to Enhanced Retrieval Augmented Generation (RAG) with Cohere Rerank, Part 4
Our dedication to delivering top-tier products drove us to integrate Hybrid Search from Weaviate and the Cohere Rerank into our existing Ask Astro system.
Incident Management at Astronomer: 1 Year Later
One year ago, we rolled out a new incident management process. Read on to find the past, present, and future of incident management at Astronomer.
Reliable Data Orchestration for AI Applications
Discover how Dosu leverages Astronomer to streamline data orchestration for AI applications, ensuring reliable pipelines and boosting productivity. Learn how this partnership enhances AI development and supports open-source communities.
Tracking Innovation: How Astronomer Streamlined Release Notes with Towncrier
Discover how we created an efficient release note system with Towncrier. Minimized friction and improved developer workflows.
Maximizing Data Workflow Efficiency: The Advantages of Using Airflow with Azure Data Factory
ADF excels in creating quick and low-code data jobs with an intuitive UX. By layering in Airflow’s expressiveness orchestration capabilities on top of ADF workflows, developers get end-to-end visibility of their workflows without needing to migrate any jobs.
Welcome to our new New York City headquarters!
Astronomer has moved! At the start of this year, we relocated our headquarters to the heart of New York City at 50 West 23rd Street to support our growing business and customer base
How Astro runs billions of Airflow tasks around the world
A walkthrough of Astro's architecture and deployment model
Astro by Astronomer Delivered 438% ROI: Insights from a Forrester TEI Study
The value of Astro: a summary of Forrester’s Total Economic Impact™ Study
Standardizing your Astro projects with Cookiecutter and Cruft
A demonstration of how a platform team can develop a template Astro project for bootstrapping Astro projects for development teams. We demonstrate how to use Cookiecutter for developing a template project and Cruft for synchronizing generated projects with changes in the template project.
Revolutionizing Data Orchestration and MLOps with Apache Airflow® and Astronomer at Chiper
In this post we will cover how we had implemented a solid MLOps pipeline and data orchestration with the help of Airflow in multiple use cases.
Introducing the Astronomer Champions Program for Apache Airflow®
Today, we're thrilled to announce the launch of the Astronomer Champions Program for Apache Airflow®, a global initiative designed to recognize and empower outstanding data practitioners who are dedicated advocates of this powerful open-source orchestration tool.
Introducing Airflow 2.8
The latest minor Airflow release includes new features and improvements such as the Airflow ObjectStore, Listener hook for Datasets, enhanced logging capabilities, and more.
Deploy Rollbacks: Upgrade Airflow and Deploy DAGs with Confidence
The new Deploy Rollbacks feature enables users to revert any code deployed to Astro Deployments, including upgrades, to a known "good" state. This allows users to quickly recover from failing pipelines and avoid critical downtime.
Advanced XCom Configurations and Trigger Rules Tips and Tricks to Level-Up your Airflow DAGs
Master Airflow's Trigger Rules & XComs for flexible, resilient data pipelines. Learn how to handle complex scenarios and ensure flawless workflow execution.
Introducing the Astro Platform Release, Q4 2023
Unveiling Astro's latest features for streamlined connectivity, confident upgrades, and cost-efficient scaling. In this article, we’ll dive into these key features and explore how they can benefit your organization.
Enhanced Authentication Security to your Data Services on Azure with Astro
Experience advanced authentication with Apache Airflow®™ on Astro, the Azure Native ISV Service. Securely orchestrate data pipelines using Entra ID. Follow our step-by-step guides and leverage open-source contributions for a seamless deployment experience.
Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools
Build production-ready ML applications with Airflow's integrations for LLMs and AI.
Introducing Apache Airflow® on Astro – an Azure Native ISV Service
Introducing Apache Airflow® on Astro, an Azure Native ISV Service. This partnership with Microsoft seamlessly embeds Apache Airflow® into the Azure ecosystem, offering a unified environment for scalable, secure, and easy-to-manage mission-critical data pipelines.
Apache Airflow® TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs
Explore the TaskFlow API and traditional operators and find out how to combine them for dynamic, efficient DAGs.
Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider
See how the new Tecton Airflow Provider can make your feature pipeline orchestration within Apache Airflow® more efficient.
Ask Astro: Operationalizing Data Ingest for Retrieval Augmented Generation with LLMs, Part 3
Data ingestion for RAG with LLMs: Ask Astro Part 3 covers vector stores, schema design, and chunking.
Using Astronomer’s new Cosmos to deploy dbt pipelines onto Snowflake
The state of deploying pipelines with dbt has changed considerably in the last few months. Over the last few weeks, I was working with Astronomer to test out their new tool, Cosmos, to deploy dbt workflows onto Snowflake.
Databricks vs. Airflow From a Management Perspective, Part 2
Databricks vs Airflow from a production management perspective. Explore the differences in setup, monitoring, integrations, scalability & customization.
ML for Customer Analytics with Airflow, Snowpark, and Weaviate
An example project showing how to use Apache Airflow® to orchestrate a machine learning pipeline with the Snowpark provider and Snowpark ML.
Migrate Python Jobs to Airflow in 4 Simple Steps
Learn how easy it is to migrate Python scripts to Airflow DAGs, streamline orchestration, and leverage Airflow features to boost job efficiency.
3 Key Takeaways from Airflow Summit 2023
Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.
Ask Astro: An open source LLM Application with Apache Airflow®, Part 2
Build an LLM-powered chatbot with Airflow! Learn how to leverage domain-specific knowledge to create intelligent applications like "Ask Astro." Astro LLM meets Apache Airflow.
Day-2 Operations for LLM Applications with Apache Airflow®
Explore how Apache Airflow® solves day-2 operations challenges for LLM applications. Learn about scalable, reliable, and auditable workflows with Airflow.
Advanced Airflow CDC Implementation
Dive deeper into Airflow CDC implementation. Explore advanced use cases, best practices, and handle schema evolution & log-based sync effectively.
Comparing Data Orchestration: Databricks Workflows vs. Apache Airflow®, Part 1
Choosing the right data orchestration tool for your needs can be tough. This blog post compares Databricks Workflows and Apache Airflow, two popular options.
Debugging Airflow Made Easy: 3 Key Steps to Debug your DAGs
Learn how to debug Airflow DAGs in 3 key steps. Eliminate common issues, set up a local development environment, and implement testing for seamless Airflow workflows.
Orchestrating Machine Learning Pipelines with Airflow
In the realm of machine learning, managing workflows efficiently is paramount. One tool that has emerged as a game-changer in this space is Apache Airflow®.
Change Data Capture (CDC) in Airflow: A Beginner's Guide
Understand the basics of Change Data Capture (CDC) in Airflow. Learn its importance, use cases, and core concepts for data pipeline success.
Introducing Airflow 2.7
The latest minor release includes several new features, such as automatic setup/teardown of tasks, built-in OpenLineage support, cluster activity view, fail-stop functionality, and more.
Leveraging Apache Airflow® and Kubernetes for Data Processing
Get insights on how to use Apache Airflow® and the Kubernetes Executor for data processing, along with proven best practices and tips for scaling your workloads.
Test Airflow Upgrades with the Astro CLI
The Local Upgrade Test command in the Astro CLI eliminates upgrade pains and ensures safe upgrades, allowing users to confidently identify and resolve compatibility issues, and DAG import errors.
Enhanced Astro Workspace Roles for more granular permissions
In this blog post, we will dive into the details of the Astro’s Role-Based Access Control (RBAC) and new Workspace Role updates, and explore improvements to popular use cases of Astro.
ETL in Airflow: A Comprehensive Guide to Efficient Data Pipelines
Optimize your data pipelines with Apache Airflow®. This guide covers tips for faster, more reliable, and easier-to-manage ETL workflows.
Run ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud
Use Apache Airflow® with CrateDB to run ETL processes, deploy with ease thanks to Astro and CrateDB Cloud.
Introducing Cosmos 1.0: the best way to run dbt Core in Airflow
The easiest way to orchestrate dbt Core using Apache Airflow®
6 Lessons Learned in Building Astronomer’s Developer Documentation
The Astronomer Approach to Clear and Effective Technical Documentation
Airflow Monitoring: Mastering SLAs, DAGs, & Observability
Learn how to effectively monitor Airflow DAGs, track SLAs, and maintain data pipeline health. Explore Airflow UI, notifications, and advanced observability tools.
.jpg?_cchid=bff73ff9251b57e4bd317661c891cb2f)
Advantages of Hosted Airflow for Your ETL Workflows
Maximize ETL efficiency with hosted Apache Airflow® on Astro, not self-hosting open-source. Benefit from simplified infrastructure management, scalable elasticity, and dedicated support for your workloads.
Best Practices for Building an Airflow Service (Part 1)
This guide covers best practices for everything from choosing the right Airflow deployment model to configuring your DAGs for optimal performance.
Astronomer and Fivetran Partner to Release Production-Grade ELT Airflow Provider
The new provider will make it easier for organizations to use Airflow to automate and manage their Fivetran pipelines.
Astronomer and Snowflake: Unleash the Power of Snowpark Container Services and Apache Airflow®
Announcing the Astronomer and Snowflake partnership! Transform your data pipelines with Snowpark and Airflow.
Three ways to use Airflow with MotherDuck and DuckDB
Use Apache Airflow® with DuckDB and MotherDuck in three different ways. Access the DuckDB Python package directly, leverage the DuckDB Airflow provider, and use DuckDB with the Astro Python SDK.
The Astro Cloud IDE: from Python and SQL to nearly 1,000 Airflow operators
Use ~1,000 open-source Airflow operators and define your own custom operators in the Astro Cloud IDE with the newly released cell type functionality
The Top 7 Alternatives to MWAA
Beyond MWAA: Top 7 data orchestration tools for optimized workflows and increased productivity.
The Top 7 Alternatives to Google Cloud Composer
Picking the right tools for your data stack depends on your exact business and engineering needs, and the choice may seem daunting. Thankfully, there are several popular tools, each with thousands of users, all with a unique approach for managing data pipelines.
How we optimized the Registry for performance across millions of page views
Migrating the Astronomer Registry’s backend from Airtable to Postgres and a Golang REST API
Scale Airflow with confidence using Astro’s new Alerting capabilities
This new functionality in Astro lets you easily implement DAG-level or task-level alerts to be sent to Slack or PagerDuty.
Introducing Airflow 2.6
Apache Airflow® 2.6 contains over 500 commits from over 130 contributors, adding up to 35 new features, 50 general improvements, and 27 bug fixes.
Kubernetes Executor Support in Astro: A New Era of Scalability and Resource Management
Announcing Kubernetes Executor support in Astro. You can now take advantage of the power of Kubernetes to manage resources & scale your Airflow workloads.
Astro CI/CD Enforcement for Code Changes
Ensure that all code changes are deployed within your CI/CD processes, increase code quality, and enforce automated testing.
OpenLineage Is on the Rise in 2023
Learn why OpenLineage is catching on and see what lies ahead for this open-source standard.
Inside Authorized Workspaces, A New Feature in Astro
Authorized Workspaces, a new feature in Astro, lets customers isolate teams or projects to specific clusters in their data planes.
Introducing Support for the Kubernetes Executor on Astro, Now in Private Preview
The Kubernetes Executor offers Astro customers task isolation, efficient resourcing, and simplicity.
Improving a Data Quality Process by Adding Great Expectations
Simplify data quality checks in Airflow with Great Expectations. Learn how to integrate, set up, and leverage its powerful features for reliable pipelines.
Introducing Astro’s New Workspace Homepage
The updated Astro homepage brings together the key pieces of information a user needs to start their day.
The Airflow Year in Review 2022
Find out how Airflow has been optimized in 2022. Learn about major updates, including data-driven scheduling, dynamic task mapping, and UI enhancements.
Win a Scholarship for CoRise’s “Effective Data Orchestration with Airflow” Course
Learn how to win a full scholarship to CoRise’s new Airflow and data orchestration course.
The New, Faster Way to Deploy Airflow DAGs to Astro
The new DAG-only deploy feature in the Astro CLI makes deploys to Astro significantly faster and allows for more flexibility in CI/CD workflows.
How an Improved DAG-Testing Command in the Astro CLI Made Its Way into Airflow
Learn how improved DAG-testing commands in the Astro and Airflow CLIs make DAG authoring easier and help DAGs run more reliably.
5 Ways to View and Manage DAGs in Airflow
Find out what the most popular and useful DAG views in the Airflow UI are. Learn about the Airflow Graph View, Grid View, Calendar View, and Browse Tab.
Introducing the Astro Cloud IDE
Discover the Astro Cloud IDE, a notebook-inspired tool for writing data pipelines. See how to define tasks and connections without knowing Apache Airflow®.
What’s New in Apache Airflow® 2.5
Check out what’s new in Apache Airflow® 2.5. Learn more about improvements to Airflow’s dynamic task mapping and data-dependent scheduling features.
A Short History of DAG Writing
Learn about Airflow & its updated features. Get to know how users can benefit from Taskflow API, Custom XCom Backends, Astro SDK, and the Astro Cloud IDE.
Best Practices for Secure Network Connectivity and Authentication in Astro
Learn how to securely hook up data sources and implement strong authentication in Astro — a modern data orchestration solution powered by Apache Airflow®.
3 Ways to Extract Data Lineage with Airflow
Learn how to extract data lineage events from your Airflow pipelines using OpenLineage. Plus, see how these three methods work with the Astro platform.
How Astro’s Data Graph Helps Data Engineers Run and Fix Their Pipelines
Discover how Astro can help you understand, communicate, and solve pipeline problems. Learn about the key pipeline observability feature – the Data Graph.
OpenLineage: Where It Came from and What Comes Next
Hear from Julien Le Dem, Chief Architect at Astronomer, about the creation of OpenLineage and how it’s evolving into a standard for data lineage.
How to Keep Data Quality in Check with Airflow
Learn more about Airflow-driven data quality checks, their benefits, and design. Find out how data quality issues are detected and solved at Astronomer.
What’s New in Astro Python SDK 1.1: Data-Driven Scheduling, Dynamic Tasks, and Redshift Support
Learn what the new upgraded Astro Python SDK 1.1 offers to Airflow users. Find out more about data-driven scheduling, dynamic tasks, and Redshift support.
Orchestration, or How to Become a Data-Driven Company
Learn how Astro, the modern Airflow-powered data orchestration platform, helped Astronomer build a fully coordinated data ecosystem.
Micropipelines: A Microservice Approach for DAG Authoring in Apache Airflow®
Learn how to tune the data system and have critical data products ready on time with micropipelines. See how to make DAG authoring easier with Airflow 2.4.
Expanding Data Access and Exchange Inside a Company
Hear from Steven Hillion and Taylor Merrick about how our data scientists combine tooling and process to encourage company-wide data product development.
Airflow 2.4 and Data-Driven Scheduling: How a New Feature Is Saving Time at Astronomer
Learn how Astronomer is using the new data-driven scheduling feature in Airflow 2.4. See how it benefits DAG authors and helps solve timing problems.
Astro CLI: The Easiest Way to Install Apache Airflow®
Learn all about the Astro CLI — the free, open source tool that makes it easy to install, run, and test Apache Airflow® from your command line.
Apache Airflow® 2.4 — Everything You Need to Know
Discover the newly released Apache Airflow® 2.4. Find out how its new data-driven scheduling logic enables faster and easier delivery of data.
Podcast Spotlight: What Observability Brings to Data Orchestration
Hear from Astronomer’s Senior Vice President of R&D about how data orchestration and observability improve the quality and reliability of dataflows.
Announcing Astro’s HIPAA and PCI-DSS Compliance
Two more reasons that users who need high levels of data security and protection can count on Astro.
How We Track the Growth of Apache Airflow®
Find out how we use data to keep track of what’s happening in the Airflow project.
Reimagining Airflow for Data Engineers and Data Scientists with the Astro Python SDK
Discover the Astro Python SDK—an open-source framework for writing Airflow data pipelines.
Astro Is Now Available on All Major Cloud Providers
Astro is now available on AWS, Microsoft Azure, and Google Cloud in 47 regions across six continents. Learn more on our blog.
Everything You Should Know About Airflow 2.3’s New Grid View
Learn about Airflow 2.3’s new grid view. Find out how to easily visualize complex representations in Airflow’s UI with this long-awaited intuitive feature.
The Astronomer Providers Package — A Better Option for Long-Running Tasks
Learn about Astronomer Providers — a collection of open-source operators, hooks, and sensors that allow you to schedule long-running tasks asynchronously.
Ventana Research Names Astronomer as a Finalist for Its Digital Innovation Awards
Astronomer is a finalist for the 15th Annual Ventana Research Digital Innovation Awards, recognized for innovative technologies in their markets.
Introducing Astro, the Fully Managed Data Orchestration Platform, Powered by Airflow
Discover Astro – the data orchestration platform powered by Airflow. Find out how to build, run, and observe data pipelines efficiently and with context.
Introducing New Astro CLI Commands to Make DAG Testing Easier
Learn how the new Astro CLI commands provide a great DAG development experience for Airflow users. Get to know the dev parse and dev pytest commands.
To Build or to Buy? DIY Orchestration with Airflow vs. A Fully Managed Service
Hear some reasons organizations consider building their own Apache Airflow® infrastructures, and how a fully managed service makes you more competitive.
Introducing Astronomer Providers
Find out more about Astronomer Providers, a set of Airflow 2-licensed providers with async functionality, created and maintained by Astronomer experts.
Apache Airflow® 2.3 — Everything You Need to Know
Learn what’s new in the latest release of Apache Airflow® 2.3 and how it can improve data orchestration.
Apache Airflow® for Data Scientists
Learn all about the role and challenges of a data scientist and find out how Apache Airflow® can help with your workflows.
10 Best Practices for Modern Data Orchestration with Airflow
Learn best practices for standing up, scaling, and growing Apache Airflow® to support modern data orchestration.
Airflow and dbt, Hand in Hand
Learn how to use Airflow and dbt together to advance data orchestration and data transformation projects and facilitate collaboration across data teams.
What Is Data Lineage and Why Does It Matter?
Discover the power of data lineage and its role in improving data observability and quality. Find out how to take data orchestration to the next level.
Astronomer Acquires Datakin, the Data Lineage Tool
Learn how Astronomer acquired Datakin, the real-time, operational data lineage tool from the founders of the OpenLineage and Marquez open-source projects.
Letter from the CEO: Our Story So Far
Hear how Joe Otto reflects on Astronomer’s history, and looks to a future powered by the combination of orchestration, lineage, and observability.
TechCrunch on Astronomer’s Big News
The site covered our recent acquisition of Datakin and our Series C round.
Apache Airflow® for Data Leaders — How to Empower Data Teams
Learn the common challenges data and analytics leaders face and how they use Apache Airflow® and Astronomer to empower themselves and their data teams.
Airflow Summit 2022 — Join the Airflow Event of the Year!
The biggest community-driven event around Apache Airflow® returns May 23–27, 2022.
Apache Airflow® at Astronomer—Taking Data Orchestration to the Next Level
Learn how Astronomer became the top data orchestration platform based on Apache Airflow®. See how to apply Airflow in your ETL and analytics use cases.
Airflow Best Practices
Master Apache Airflow® with these 10 best practices. Learn how to optimize your data pipelines, improve efficiency, and avoid common pitfalls.
Top Data Management Trends for 2022
Learn about emerging trends that are revolutionizing the world of data from the leading Apache Airflow® experts. See how to efficiently manage data in 2022.
Adding Data Quality to DAGs ft. Great Expectations
Find out how to use Great Expectations in an Airflow Directed Acyclic Graph to successfully perform, prioritize, and schedule data quality checks.
Astronomer and Uturn Partner to Drive Innovation and Better Business Outcomes
Learn more about Astronomer’s new partnership with Uturn Data Solutions – the leading experts in enterprise cloud enablement and application modernization.
Apache Airflow® for Data Engineers—How to Leverage Data Orchestration
Gain insight into the role and responsibilities of a data engineer. See 7 examples of how Apache Airflow® can make data engineering less challenging.
How to orchestrate Azure Data Explorer queries with Airflow
Learn how to orchestrate Azure Data Explorer queries with Airflow.
How to Select the Best ETL Tool to Integrate With Airflow? Our 3 Picks
Learn from our experts about how to select the best ETL tool, and why it pays to integrate Fivetran, Airbyte, and Azure Data Factory with Apache Airflow®.
Every Company Nowadays Becomes a Data Company—Interview with Bolke de Bruin
Hear from Bolke de Bruin – VP of Enterprise Data Services at Astronomer – about how Apache Airflow® helps modern companies manage data effectively.
Machine Learning Pipeline Orchestration
Learn about machine learning orchestration, machine learning pipelines, and their components. See why Apache Airflow® is the top ML data orchestration tool.
How to Build a Modern Data Stack
Breaking down what a modern data stack means in practice. We discuss four core components, five reasons to set it up, and how to orchestrate it.
Apache Airflow® vs. Apache Beam: A Comparative Guide
Explore the differences and similarities between Apache Beam and Airflow. Understand their capabilities, programming models, and ideal use cases to make the right choice for your data management needs.
Democratizing the Data Stack—Airflow for Business Workflows
See why modern orchestrators and reverse ETL tools are the future for data-driven business. Learn how Apache Airflow® takes SQL to the next level.
Machine Learning Pipeline: Everything You Need to Know
Discover what a machine learning pipeline is and the process behind creating one with Apache Airflow®. Learn what you need to know about ML pipelines.
What is Reverse ETL and How Can It Improve Data Flow?
Understand the difference between ETL and reverse ETL. Learn to use the Census reverse ETL platform and Airflow together to leverage data orchestration.
Airflow at BBC—Data Orchestration Solution in Media
Hear what the BBC's Data Engineer says about the popularity of orchestration tools in the media industry. Find out why the BBC went for Apache Airflow®.
Everything You Need to Know About Apache Airflow® 2.2.0
See what’s new in Apache Airflow® 2.2. Learn about big improvements, bug fixes, and internal changes, and the benefits they bring to Apache Airflow® users.
Big Data Architecture: Core Components, Use Cases, and Limitations
Learn more about the concept of Big Data Architecture and its 5 core components. See how various companies benefit from implementing Big Data Architecture.
The Future of Banking: How Can Apache Airflow® Help?
Learn how to make the most of data in banking. See which banks use Airflow, and how the top orchestration tool helps them overcome major challenges.
Apache NiFi vs. Apache Airflow®
Get to know the major benefits and limitations of Apache NiFi and Apache Airflow, and see which of the two popular ETL tools is better for data management.
Airflow at Wise: Data Orchestrator in Machine Learning
Alexandra Abbas—a Machine Learning Engineer at Wise—explains what makes Airflow an ideal tool for data orchestration in the fintech industry.
How to Build an ETL Process?
Learn what an ETL process is and how to build it. Find out how Apache Airflow® can help you create, scale, and manage ETL pipelines more effectively.
Data Silos: What Are They and How to Fix Them?
Find out what causes data silos and how they hurt your business. Learn how Airflow and data orchestration can solve the data silo problem in your company.
Airflow at Societe Generale: Data Orchestration Solution in Banking
Hear from the Product Owner at Societe Generale about the benefits of implementing Airflow. Find out how data orchestration solutions are used in banking.
Data Pipeline: Components, Types, and Best Practices
Learn the basics of data pipeline building. Get to know data pipeline components, types, and best practices. See how Airflow can simplify the process.
Generating Airflow DAGs from dbt Models (Part 3)
Streamline DAG generation! Explore a utility package to create Airflow DAGs from dbt models. Get sample configurations & customizable code.
What Is Data Orchestration and Why Is It Essential for Business
Data orchestration is the process of collecting siloed data from multiple locations and systemizing, unifying, and activating it for data analysis.
Airflow Summit 2021 Highlights
Learn what the Airflow community got up to in 2021, in this recap of the biggest international Airflow event. Get ready for the next Airflow Summit!
How Data Pipelines Drive Improved Sales in E-commerce
Hear from Viraj Parekh–Field CTO at Astronomer–about how data pipelines help increase online sales. See why Apache Airflow® works for e-commerce companies.
Everything You Need to Know About the Airflow Summit 2021
Learn why it’s worth attending the biggest Airflow conference for developers and data professionals. Check what’s on the agenda and register for free.
Validate Your Apache Airflow® Skills With the Astronomer Certification
Find out how to get started with Apache Airflow® and enhance your knowledge. Learn essentials from Astronomer’s experts and become Apache Airflow®-certified!
The New KubernetesExecutor
Learn more about the KubernetesExecutor and its upgrade to version 2.0. See new features redesigned with Airflow admins and data engineers in mind.
How to orchestrate Talend jobs with Airflow
Learn how to orchestrate Talend jobs with Airflow so you can use both tools without rewriting your pipelines.
Announcing the Astronomer Registry
Learn about the discovery-and-distribution hub for Airflow integrations. See how to bridge the gap between the Airflow community and the data ecosystem.
Airflow 2.0 TaskFlow API and Its Features
Learn more about the TaskFlow API and read about its features. Get to know how TaskFlow API in Airflow 2.0 enables a better DAG authoring experience.
Airflow Secrets Management: Best Practices for Airflow 2.0
Have a look at Astronomer’s ultimate guide on Airflow Secrets, and learn best practices for managing Secrets with various backends in Apache Airflow® 2.0.
Near-Real-Time CDC with Airflow and GCP
Learn how to implement near-real-time Change Data Capture (CDC) in Airflow using a scheduled GCP CloudSQL export approach for data pipelines.
Deploying Airflow DAGs in Production with dbt (Part 2)
Take your Airflow DAGs live! Learn how to deploy them in production using dbt manifest.json, and integrate dbt into your ETL/ELT workflows.
Building a Scalable Analytics Architecture With Airflow and dbt (Part 1)
Kickstart your analytics architecture with Airflow and dbt. Learn DAG authoring, configurations, and code snippets for a seamless setup.
The Airflow 2.0 Scheduler
Explore the features of the updated Apache Airflow® 2.0 Scheduler. Learn how the Airflow Scheduler enables quick and seamless initiation of tasks.
A Great Expectations Provider for Apache Airflow®
Find out why Great Expectations and Apache Airflow® are a great match. Learn how to leverage native Great Expectations functionality directly from DAGs.
Introducing Airflow 2.0
Get to know the highlights of Apache Airflow® 2.0 and see hundreds of new features it includes. Have a look at how Airflow 2.0 compares to Airflow 1.10.
Introducing KEDA for Airflow
Explore the possibilities of the Kubernetes Event-Driven Autoscaler. See how KEDA helps users improve the efficiency of their Apache Airflow® deployments.
Profiling the Airflow Scheduler With Flame Graphs
Get tips on improving the performance and reliability of the Airflow Scheduler. Find out how to benchmark and profile it using py-spy and Flame Graphs.
Why Airflow?
Discover Apache Airflow® and explore its workflow-management capabilities. See which global companies use Airflow to solve data engineering challenges.
The Next Generation of Astronomer Cloud
Learn how Astronomer Cloud supports the latest version of Apache Airflow®. See the features included in the newly released, next-generation data platform.
Astronomer v0.10
Learn more about Astronomer v0.10 and its key updated functionalities. See how the new Astronomer Platform supports the latest version of Apache Airflow®.
7 Common Errors to Check When Debugging Airflow DAGs
Get to know best practices for debugging Apache Airflow® DAGs. Check out the list of common Airflow deployment errors, and see how to find and remove them.
Airflow Design Principles: Multi-tenant vs. Monolithic Architecture
Find out how to design an Airflow infrastructure, and whether it makes more sense to power your DAGs with one monolithic Airflow instance, or many.
Astronomer v0.8.0 Release Notes
Discover the newly launched Astronomer v0.8.0 and the features it includes. Find out what’s been fixed, improved, and added to the Astronomer Platform.
Astronomer on Astronomer: Loading Thousands of Files Into Redshift With Apache Airflow®
Get to know why Astronomer switched to Apache Airflow®. Learn how we optimized Airflow to fit our initial needs and what features we're planning to build.
Astronomer v0.7.0 Release Notes
Release notes covering the features released with v0.7.0 of the Astronomer platform.
Astronomer v0.6.0 Release
Release notes for v0.6.0 of the Astronomer platform.
Astronomer v0.5.0 Release
Release notes from our recent platform update to v0.5.0.
Astronomer v0.4.1 Release
Check out the highlights of the Astronomer v0.4.1 release. See the full summary of upgrades and learn more about the Astronomer Platform's new features
How the Apache Airflow® Project Will Change
Explore the future of Apache Airflow® at Astronomer. Take a look at the official Airflow roadmap and see what improvements and developments to expect.
Astronomer v0.3.2 Release
Learn more about Astronomer v0.3.2 and its new, updated functionalities. Find out what’s been changed, fixed, and added to the Astronomer Platform.
Announcing Astronomer v0.3
Find out more about Astronomer v0.3 and its great benefits. Get to know what features are included in the newly released next-generation data platform.
Announcing the Astronomer Platform, a Managed Service for Apache Airflow®
Learn about the benefits of the Astronomer’s Managed Service for Apache Airflow®. See why our experts decided to design and build the Astronomer Platform.
Announcing Astronomer SpaceCamp
Discover Astronomer SpaceCamp and see how it gets data teams up and running with Airflow in no time. See the benefits of different SpaceCamp versions.
Announcing The Airflow Podcast
Learn about Astronomer's podcast focused on the future potential of Apache Airflow®, as seen by top players in the data engineering space.
An Airflow Story: Cleaning and Visualizing Our Github Data
Check out instructions on importing and moving Github data using Apache Airflow®. See how to deal with Github DAG writing, visualization, and dashboarding.
Improving Government Services With Apache Airflow®: a Q&A With San Diego’s Chief Data Officer
Hear from Maksim Pecherskiy–Chief Data Officer of the City of San Diego–about how Apache Airflow® helps operationalize data in the public sector.
From Behavioral Analytics to Data Science With Astronomer
Find out how to scale behavioral analytics with Apache Airflow®. See why the Astronomer Platform is an ideal solution for data scientists and analysts.
Using Apache Airflow® to Create Data Infrastructure in the Public Sector
Learn why ARGO chose Apache Airflow® to build and maintain data infrastructure. Find out how they transform basic public services with Airflow.
Data Format Types: CSV, JSON, & XML Demystified
Explore common data format types (CSV, JSON, XML) & understand their pros, cons, & ideal use cases. Learn how to choose the right format for your data needs.
Why Every Data Scientist Needs a Data Engineer
Get to know the roles and responsibilities of data scientists and data engineers. Learn why data engineering and data science go hand in hand.
What Exactly Is a DAG?
Learn what a DAG is and how it's used in data pipelines. Explore benefits, real-world examples, and FAQs in this comprehensive guide.
Data Engineering Platform Astronomer Closes $3.5M Financing
Learn how Astronomer, a data engineering platform, closed $3.5M in new financing, led by Wireframe Ventures in San Francisco and CincyTech in Cincinnati.
Normalizing Data for Warehouse Centralization
Knowing the options for storing data will help you make the right decisions for your company when you’re ready to take this step.
Apache Airflow® and the Future of Data Engineering: A Q&A with Maxime Beauchemin
Hear from Maxime Beauchemin – a data engineer at Airbnb and creator of their data pipeline framework, Airflow – about the future of data engineering.
Our Open Source Philosophy
Get to know how the open-source approach helps drive growth and innovation. Learn why it’s worth investing in open-source components like Apache Airflow®.
Why Is My Data Playing Hard to Get?
Learn more about the different types and properties of hard-to-reach data with great potential. Find out how to access, organize, and store it effectively.
Airflow at Astronomer
Learn why Astronomer needed a unified scheduling system to extract and monitor all types of data pipelines. Find out why Apache Airflow® was our answer.
Press Release: Astronomer Announces Seed Financing
Press Release: Astronomer Closes $1.9M in Seed Financing
Our Unique Path to Raising $2M Seed in the Midwest
Learn about Astronomer’s 2-year journey to raising $2 million seed, including the ups and downs we went through to take our company to the next level.
Lessons Learned Writing Data Pipelines
See how to simplify the data pipeline writing process with the right tools. Learn what Astronomer experts do to make data pipelines less challenging.
Why We Built Our Data Platform on AWS, and Why We Rebuilt It With Open Source
Find out how Astronomer leveraged AWS services with open-source alternatives. See why Airflow and Apache Mesos help build more and better integrations.
An Almost Acquisition Story
Coming out of AngelPad’s 2015 Demo Day, we found ourselves vacillating between an acquisition and Series A, though we were arguably too early for either.
Announcing Astronomer v0.9
Release notes for v0.9 of the Astronomer Platform
A Logo Story
Astronomer's Head of Design, Chris Hendrixson, explains how he created the design aesthetic to encompass data, futurism, and a little bit of fun.
Setting Up Your Redshift Cluster
Redshift is popular but you still need to know what you''re doing when spinning up your first cluster. In this tutorial, we walk you through the process.
When Should You Start to Warehouse Your Data?
Learn how to leverage your business data with data warehousing. Discover the best time to create a data warehouse, and see which warehousing tools to use.
Why We Drove to NY and Back Over the Past 48 Hours for a 15-Minute Meeting
Find out why it was worth driving from Cincinnati to New York for a fifteen-minute meeting with the number one ranked startup accelerator in the world.
