Astronomer Blog

The latest insights from our team of Apache Airflow experts.

November 28, 2023

Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools

Steven HillionSVP of Data and AI

Apache Airflow is at the core of many teams’ ML operations, and with new integrations for Large Language Models (LLMs), Airflow enables these teams to build production-quality applications with the latest advances in ML and AI.

November 15, 2023

Introducing Apache Airflow™ on Astro – an Azure Native ISV Service

Introducing Apache Airflow™ on Astro, an Azure Native ISV Service. This partnership with Microsoft seamlessly embeds Apache Airflow into the Azure ecosystem, offering a unified environment for scalable, secure, and easy-to-manage mission-critical data pipelines.

November 2, 2023

Apache Airflow TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs

Explore the TaskFlow API and traditional operators and find out how to combine them for dynamic, efficient DAGs.

October 31, 2023

Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider

See how the new Tecton Airflow Provider can make your feature pipeline orchestration within Apache Airflow more efficient.

October 24, 2023

Ask Astro: Operationalizing Data Ingest for Retrieval Augmented Generation with LLMs, Part 3

In part 3 of Ask Astro series, uncover key considerations for data ingestion in Retrieval Augmented Generation with LLMs. Find out how to select the ideal vector store, model, schema design, and chunking strategy for your project.

October 12, 2023

Using Astronomer’s new Cosmos to deploy dbt pipelines onto Snowflake

The state of deploying pipelines with dbt has changed considerably in the last few months. Over the last few weeks, I was working with Astronomer to test out their new tool, Cosmos, to deploy dbt workflows onto Snowflake.

October 11, 2023

Databricks vs. Airflow From a Management Perspective, Part 2

Part 2 of our blog series offers an in-depth comparison between Databricks and Airflow from a management perspective. Explore the differences in setup, monitoring, integrations, scalability, and customization.

October 6, 2023

Migrate Python Jobs to Airflow in 4 Simple Steps

Learn how easy it is to migrate your Python scripts into Directed Acyclic Graphs (DAGs) with modular tasks, orchestrate them in Apache Airflow, and use Airflow features to enhance the efficiency of your jobs.

September 29, 2023

3 Key Takeaways from Airflow Summit 2023

Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.

September 22, 2023

Ask Astro: An open source LLM Application with Apache Airflow, Part 2

A reference implementation and community dialogue for building production-grade day-2 operations for LLM applications

September 21, 2023

Day 2 Operations for LLMs with Apache Airflow: Going Beyond the Prototype, Part 1

We discuss the challenges of LLM application development beyond the prototype and why Apache Airflow was highlighted for its ability to enable day-2 operations for LLM applications.

September 8, 2023

Change Data Capture with Airflow - Part 2

We look at some examples to implement CDC in Airflow using a legacy data pipeline with batch processing, and look at advanced use cases for near-real-time processing.

Astronomer Blog

Ready to Get Started?