Skip to main content
Astro supports a range of options when it comes to organizing your project code. This guide covers the options along with their pros and cons, so you can choose the best option for your team.

​
Feature overview

Astro supports three basic repository strategies. You can:
  1. Use a monorepo: host an Astro project in one repository for deploys to one or multiple Astro Deployments, based, for example, on permanent dev and prod branches with their own Deployments or a single permanent branch. Astronomer recommends this approach for most teams.
  2. Use a multirepo: separate your Dags from other files in your project and maintain multiple repositories linked to one or multiple Astro Deployments. Astronomer recommends this approach for teams needing to meet strict security requirements and teams planning to automate the creation of Deployments.
  3. Use Dag bundles: let multiple teams each keep their own repository and deploy independently to a single, shared Deployment. Astronomer recommends this approach for multiple teams that want to share a Deployment without sharing a repository.

​
Best practice guidance

Depending on your organization’s structure and needs, Astronomer recommends either a monorepo or multirepo approach to organizing the code you deploy to Astro. In addition to a repository strategy, Astronomer recommends implementing version control and CI/CD.

​
Option 1: Monorepo

This is the most common strategy.
When your Dags are critical to your business, the ability to test your Dags is critical, as well. Using a monorepo, you can set up automated deploys from multiple branches of a single repository to support multiple Deployments of the same codebase, enabling implementation of robust testing of the code you intend to deploy to production, support for multiple teams, and more. Within a monorepo, you can also maintain multiple Astro projects, all with Astro Deployments and testing. This strategy requires:
  • One Astro Workspace for your project(s).
  • One or more permanent branches in your repository, each representing an environment. Many teams use two permanent branches named main and dev.
  • One or more Astro Deployments, each representing an environment.
Pros of this approach:
  • You can test code changes to your Dags in an isolated environment on Astro before deploying the changes to production.
  • It is flexible and scalable. Mapping branches to additional Deployments allows for permanent testing environments or ephemeral sandbox Deployments for feature testing. See Manage dev Deployments on Astro. Also, if you work at a larger organization, you can deploy to different Workspaces and Astro projects to support multiple teams or organize Deployments according to specific business use cases.
  • It supports separate environment configurations. For example, your development Deployment can query a development database, and your production Deployment can query a production database without needing complex logic to switch between the two.
Cons of this approach:
  • Dependencies and environments might be more difficult to manage, to the extent that they differ from project to project.
For guidance on setting up CI/CD for deploys to multiple Deployments from a single repository, see Multiple environments.
Individuals and small teams getting started on Astro can keep their instances simple with one permanent branch mapped to an Astro Deployment.

​
Option 2: Multirepo

A common use case for multiple repositories is keeping Astro Project configuration (such as the Dockerfile, requirements.txt, or Deployment Configuration-as-Code) separate from Dag code. Astronomer recommends a multirepo approach when:
  • You have strict security requirements for who can manage Deployments.
  • You want to minimize complexity for project contributors, automate the creation of Deployments, or manage Deployments more efficiently.
  • You can set up and maintain a more complex CI/CD pipeline.
This strategy requires:
  • A Workspace for your project(s).
  • Multiple repositories.
  • One or more Astro Deployments.
  • Dag-only deploys on the target Deployment and CI/CD pipeline setup for each repository.
Pros of this approach:
  • Keeping project configuration separate from Dag code enables the use of a single repository for configuring multiple Deployments.
  • The possibility of accidental changes to Astro Deployment settings is minimized.
  • You can automate the creation of Deployments and streamline your Deployment management process.
Cons of this approach:
  • You must keep any local copies of an Astro project synchronized with multiple repositories in order to test Deployment code locally.
  • Team members might have inconsistencies in their local environments if they lack access to the configuration repository. For this reason, Astronomer recommends setting up a Dev Deployment where Dag authors can see and modify project configuration for testing purposes.

​
Option 3: Dag bundles for multiple teams

This requires an Airflow 3 Deployment. See Dag bundles for complete setup steps and requirements.
When multiple teams share a single Deployment, each team can maintain its own repository and deploy to its own named Dag bundle instead of coordinating file structure and deploys through a shared repository. Astronomer recommends this approach when:
  • Multiple teams want to share the cost and operational overhead of a single Deployment.
  • Each team already maintains its own repository and CI/CD pipeline.
  • You don’t need to split Dag code from Deployment configuration within a single team’s project. See Option 2: Multirepo for that use case.
This strategy requires:
  • One Astro Workspace for the shared Deployment.
  • One repository per team.
  • One Airflow 3 Deployment with Dag-only deploys enabled.
  • A CI/CD pipeline per repository that deploys to its own bundle, for example astro deploy --dags --dag-bundle-name finance.
Pros of this approach:
  • Teams don’t need access to each other’s repositories or CI/CD pipelines.
  • A deploy from one team’s pipeline never overwrites another team’s Dags.
  • Teams can be added to or removed from a shared Deployment without migrating code between repositories.
  • Fewer Deployments to create and maintain than one Deployment per team, though a shared Deployment needs enough resources for every team’s combined workload.
Cons of this approach:
  • Every team’s Dags run in the same Airflow environment, so they must all support the same Airflow, Python, and system package versions.
  • Upgrading Airflow or the underlying image requires coordinating with every team on the Deployment.
  • One team’s Dags can compete with another team’s for dag processor and worker resources, sometimes called the noisy neighbor problem. Worker queues can isolate compute for specific Dags.
  • Dag IDs must be unique across every team’s bundle. See Dag IDs must be unique across bundles.
To restrict which Dags each team can view or trigger in the Airflow UI, combine this strategy with Dag-level access control.
For example, a Data Platform Engineer, a Data Engineer, and an Analytics Engineer can each maintain their own repository and deploy independently to the same Deployment: The Data Platform Engineer’s repository has no dags directory, so its deploy uses --image to update only the project image without touching any Dag bundle. By default, a new non-Dag bundle is served alongside main; since Dags in finance orchestrate these dbt models with Cosmos, associate the dbt bundle with finance instead.

​
Version control and CI/CD

In addition to choosing a repository strategy suited to your use case, Astronomer recommends implementing version control and CI/CD. Implementing version control is a longstanding software development best practice that enables:
  • Simultaneous collaboration.
  • Easy review of code changes.
  • Safe and secure experimentation on new features and fixes.
  • Improved traceability and auditing.
Using CI/CD is a general software development best practice that enables automated building, testing, merging, and delivery of code. If you haven’t yet implemented CI/CD pipelines for your Deployments, benefits of doing so include:
  • Faster feedback loops.
  • Improved code quality.
  • Streamlined development workflows.
  • Consistent deployments across environments.
  • Less time spent getting code to production.
  • Enhanced collaboration.
With a version control system in place, you can set up CI/CD by following the guidance in Develop a CI/CD workflow for deploying code to Astro. The option you choose for organizing the code in your shared repository and deploying to Astro should reflect the size and needs of your team. Deployment API tokens are required for implementing CI/CD for automated deploys to Astro. Once you have set up CI/CD, Astronomer recommends enabling CI/CD enforcement so that code pushes can be completed only when using a Deployment or Workspace token.