Simplify transformations with dbt in Apache Airflow

Having set up dbt-core on Airflow before, I had to create a lot of internal jobs to achieve what this project does out of the box. With Cosmos and the open source community, it's been great to see best practices in place and how forward-thinking the project is. dbt artifacts automatically spitting out open-lineage datasets to feed into data-aware scheduling? Nice! It's been great and I know it will get better.

We conducted a PoC, running dbt with Mage.ai, dbt Cloud, GitHub Actions, and Astronomer Cosmos. In the end, our team voted for Cosmos.

Astronomer Cosmos has allowed us to seamlessly orchestrate our dbt projects using Apache Airflow for our start-up. The ability to render dbt models as individual tasks and run tests after a model has been materialized has been valuable for lineage tracking and verifying data quality. I was impressed with how quickly we could take our existing dbt projects and set up an Airflow DAG using Cosmos.

Cosmos has sped up our adoption of Airflow for orchestrating our System1 Business Intelligence dbt Core projects without requiring deep knowledge of Airflow. We found the greatest time-saver in using the Cosmos DbtTaskGroup, which dynamically creates Airflow tasks while maintaining the dbt model lineage and dependencies that we already defined in our dbt projects.

With Cosmos, we could focus more on the analytical aspects and less on the operational overhead of coordinating tasks. The ability to achieve end-to-end automation and detailed monitoring within Airflow significantly improved our data pipeline’s reliability, reproducibility, and overall efficiency.