Cut DAG Runtimes by Up to 80% with Cosmos Watcher Execution Mode

Cosmos 1.11 is generally available, bringing meaningful performance improvements and enhanced visibility features that make orchestrating dbt with Airflow more efficient. With over 21 million monthly downloads, Cosmos continues to be the open-source standard for integrating dbt transformations into your Apache Airflow® pipelines.

This release focuses on helping teams run dbt workflows faster while maintaining the task-level observability that makes using Cosmos superior to running dbt projects with the BashOperator.

Introducing Watcher Mode

The standout feature in Cosmos 1.11 is the Watcher Execution Mode, a new approach to orchestrating dbt that can reduce DAG execution times significantly. In testing with real-world dbt projects, teams have seen runtime reductions of up to 80% for certain workloads.

Traditional dbt orchestration using the Local Execution Mode in Cosmos creates overhead by spawning a new dbt process for each model, seed, or test. The Watcher Execution Mode addresses this by setting up a single dbt process and updating task states using watcher tasks in Airflow as models complete. You get the same task-level visibility without the performance penalty.

Performance improvements in Cosmos 1.11:

• Up to 80% reduction in DAG runtime for workloads with many small models
• 36% faster async execution performance
• 40% improvement in DAG parsing speeds through enhanced caching

These gains translate to faster feedback loops, lower computational costs, and the ability to refresh data models more frequently without sacrificing control or visibility.

Get Started with Watcher Execution Mode

First, upgrade Cosmos to version 1.11 for example by using pip install –upgrade or modifying the pinned version in your requirements.txt file as shown below.

astronomer-cosmos~=1.11

No changes to your dbt project required. Just update your execution mode. Enable Watcher Mode in your DAGs with a simple configuration change:

from cosmos import DbtDag, ExecutionConfig, ExecutionMode

my_cosmos_dag = DbtDag(
    # ... your config
    execution_config=ExecutionConfig(
        execution_mode=ExecutionMode.WATCHER
    ),
)

Note that as this new execution mode is currently experimental and under development there are a few limitations, see the Cosmos documentation for more information.

Multi-Project dbt Docs on Airflow 3

For teams managing multiple dbt projects, Cosmos 1.11 brings native support for browsing documentation from different projects directly in the Airflow 3 UI.

dbt Docs allow you to view additional information about your dbt project, including project lineage. Hosting dbt Docs in the Airflow UI allows you to use the same RBAC you are using for your Airflow Deployment for your dbt Docs and removes the need to set up a separate service to host your dbt Docs.

What’s New in Cosmos 1.11?

Beyond Watcher Mode and multi-project docs, this release includes:

• Expanded database support: New profile mappings for MySQL and SQL Server
• Improved async execution: Cosmos async execution mode, available for BigQuery,now runs 36% faster with optimized SQL generation
• Airflow 3 compatibility: Support for Airflow 3’s new features and improved UI
• Bug fixes and stability: Comprehensive improvements based on community feedback

Optimize Your dbt Orchestration Strategy

Want to learn more about optimizing your dbt workflows with Airflow and Cosmos? Download our comprehensive Practical Guide to Orchestrating dbt with Airflow using Cosmos to explore execution modes, performance optimization strategies, and best practices for production deployments.

Build Faster with Cosmos and Airflow

Cosmos 1.11 makes it easier to run dbt at scale with Airflow. The combination of Watcher Mode’s performance improvements and support for multi-project dbt Docs support addresses common challenges teams face when managing production dbt workflows.

These improvements work with your existing dbt projects and DAG configurations. Upgrade to Cosmos 1.11, enable Watcher Mode, and start seeing faster execution times.

Get Started Today

Upgrading is straightforward:

pip install astronomer-cosmos~=1.11

For detailed configuration options and examples, check out our documentation.

Running on Astro? Cosmos 1.11 is available now with seamless dbt deploys support and enhanced observability through Astro Observe.

Share your experience with 1.11 by opening an issue on GitHub or join the conversation in the #airflow-dbt channel on Apache Airflow Slack.