Automate preview Deployments with any CI/CD tool

A preview Deployment is an Astro Deployment that a CI/CD workflow automatically creates and deletes based on feature branches in your Git repository. The workflow creates the preview Deployment when you create a temporary feature branch and deletes it when you delete the branch. Astronomer recommends using preview Deployments if you regularly need to test a small set of dags on Astro before promoting those dags to a base, production Deployment. This lowers the infrastructure cost of Deployments that are dedicated to development and testing.

To implement this feature, you need a CI/CD workflow that:

Creates the preview Deployment when you create a new branch.
Deploys code changes to Astro when you make updates in the branch.
Deletes the preview Deployment when you delete the branch.
Deploys your changes to your base Deployment after you merge your changes into your main branch.

If you use GitHub Actions as your CI/CD tool, you can find preview Deployment templates as part of the Astronomer GitHub action in the GitHub Marketplace. This GitHub action includes sub-actions for each of these four steps. To learn more, see GitHub Actions templates for preview Deployments.

To configure your own automated workflow for preview Deployments with another CI/CD tool, use the following scripts. Each of the following four shell scripts is equivalent to the steps required to implement this feature with GitHub Actions.

Create a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you create a feature branch off of the main branch of your Astro project.

$ # Install the Astro CLI
$ curl -sSL https://install.astronomer.io | sudo bash -s
$ 
$ # Get preview Deployment name
$ DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
$ BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
$ BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"
$ 
$ # Create template of Deployment to be copied
$ astro deployment inspect $DEPLOYMENT_ID --template > deployment-preview-template.yaml # automatically creates deployment-preview-template.yaml file
$ 
$ # Add name to Deployment template file
$ sed -i "s|  name:.*|  name: $BRANCH_DEPLOYMENT_NAME}|g"  deployment-preview-template.yaml
$ 
$ # Create new preview Deployment based on the Deployment template file
$ astro deployment create --deployment-file deployment-preview-template.yaml
$ 
$ # Get the ID of the new preview Deployment
$ PREVIEW_DEPLOYMENT_ID="$(astro deployment inspect -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)"
$ 
$ # Copy connections, Airflow variables, and pools from the source Deployment
$ astro deployment connection copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
$ astro deployment airflow-variable copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
$ astro deployment pool copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
$ 
$ # Deploy new code to the deployment preview
$ astro deploy -n $BRANCH_DEPLOYMENT_NAME

Creating a Deployment from a template does not copy the connections, Airflow variables, or pools from the source Deployment, so the script copies them separately with astro deployment connection copy, astro deployment airflow-variable copy, and astro deployment pool copy.

The copy commands fail if the source Deployment is hibernating. Resume the source Deployment before you run them.

Update a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script whenever you make changes in your feature branch.

$ # Install the Astro CLI
$ curl -sSL https://install.astronomer.io | sudo bash -s
$ 
$ # Get preview Deployment name
$ DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
$ BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
$ BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"
$ 
$ # Deploy new code to the preview Deployment
$ astro deploy -n $BRANCH_DEPLOYMENT_NAME

Delete a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you delete your feature branch.

$ # Install the Astro CLI
$ curl -sSL https://install.astronomer.io | sudo bash -s
$ 
$ DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
$ BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
$ BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"
$ 
$ # Delete preview Deployment
$ astro deployment delete -n $BRANCH_DEPLOYMENT_NAME -f

Deploy changes from a preview Deployment to a base Deployment

In a Deployment preview CI/CD pipeline, you run this script when you merge your feature branch into your main branch.

$ # Install the Astro CLI
$ curl -sSL https://install.astronomer.io | sudo bash -s
$ 
$ # Deploy new code to base Deployment
$ astro deploy $DEPLOYMENT_ID