GitHub Actions templates for deploying code to Astro
The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See Deploy code with the Astro GitHub integration for setup steps.
GitHub Action templates use the Astronomer-maintained deploy-action
, which is available in the GitHub Marketplace. This action automates the deploy process and includes additional features for more complex automation workflows. Specifically, the action can automatically:
- Choose a deploy type based on the files that were changed in a commit. This allows you to use the same template for DAG deploys and image deploys.
- Test DAGs as part of the deploy process and prevent deploying if any of the tests fail. These tests are defined in the
tests
directory of your Astro project. - Create a preview Deployment to test your code before deploying to production. A Deployment preview is an Astro Deployment that mirrors the configuration of an existing Deployment.
- Allows you to choose the type of code deploys used by the automation:
- (Default)
infer
image-and-dags
dags-only
dbt
- (Default)
Read the following sections to choose the right template for your use case. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation. If you do not have access to Astronomer's deploy-action
, use the private network templates.
To learn more about CI/CD on Astro, see Choose a CI/CD strategy.
If you use GitHub Enterprise and cannot access the Astronomer Deploy Action, see Private network templates.
Deploy options
The deploy-action
includes several options for you to choose a specific type of code deploy for your CI/CD processes. See the Deploy Action README to learn more about using and customizing this action, like creating a GitHub action that can support DAG and dbt deploys.
(Default) infer
By default, the deploy-action
uses infer
, which allows it to determine the type of code deploy to use based on the types of files changed in a commit: DAG or Astro project. If you committed changes only to DAG files, the action triggers a DAG deploy. If you committed changes to any other file, the action triggers an image deploy, or image-and-dags
. This setting does not include dbt
deploy types.
image-and-dags
The image-and-dags
option enables the Deploy Action to make a full project deploy, which includes both images and DAGs.
dags-only
The dags-only
deploy option enables the deploy action to deploy only the DAGs in your Astro project's dags
directory to your Deployment.
dbt
The dbt
deploy option enables the Deploy Action to deploy dbt projects to Astro, when you provide the path to a directory in your GitHub repo that contains your dbt project.
Prerequisites
- An Astro project hosted in a GitHub repository.
- An Astro Deployment.
- A Deployment API token, Workspace API token, or Organization API token.
- Access to GitHub Actions.
Each CI/CD template implementation might have additional requirements.
If you use a self-hosted runner to execute jobs from GitHub Actions, the Astro CLI's config.yaml
file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:
- Add
ASTRO_API_TOKEN
to your repository and include a check in your GitHub workflow to verify that it exists. - Use Deployment API tokens, which are scoped only to one Deployment, instead of Workspace or Organization API tokens.
- Specify
deployment-id
ordeployment-name
in your action. For example,astro deploy <deployment-id>
orastro deploy -n <deployment-name>
. - Add the command
astro logout
at the end of your workflow to ensure that your authentication token is cleared from theconfig.yaml
file.
Default deploy action
By default, the deploy action uses the infer
deploy type, which enables the action to determine whether to use either a dags-only
deploy or an image-and-dags
deploy, depending on the files you change.
If you stage multiple commits to DAG files and push them all at once to your remote branch, the template only deploys DAG code changes from the most recent commit. It will miss any code changes made in previous commits.
To avoid this, either push commits individually or configure your repository to Squash commits for pull requests that merge multiple commits simultaneously.
Setup
- Single branch
- Multiple branch
- Custom Image
To automate code deploys to a single Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
-
Set the following as a GitHub secret:
ASTRO_API_TOKEN
: The value for your Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API credentials as environment variables
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <your-deployment-id>
The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.
Configuration requirements
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have at least one API token with access to both of your Deployments.
Implementation
-
Set the following as GitHub secrets:
PROD_ASTRO_API_TOKEN
: The value for your production Workspace or Organization API token.DEV_ASTRO_API_TOKEN
: The value for your development Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API credential as an environment variable
ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <dev-deployment-id>
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets Prod Deployment API credential as an environment variable
ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <prod-deployment-id>
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Prerequisites
- An Astro project that requires additional build-time arguments to build the Runtime image.
Implementation
- Set the following as a GitHub secret:
ASTRO_API_TOKEN
: The value for your Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
uses: docker/build-push-action@v2
with:
deployment-id: <your-deployment-id>
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <your-deployment-id>
image-name: ${{ steps.image_tag.outputs.image_tag }}For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:
name: Astronomer CI - Custom base image
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Create SSH Socket
uses: webfactory/ssh-agent@v0.5.4
with:
# GITHUB_SSH_KEY must be defined as a GitHub secret.
ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
- name: (Optional) Test SSH Connection - Should print hello message.
run: (ssh git@github.com) || true
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
ssh: |
github=${{ env.SSH_AUTH_SOCK }
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <your-deployment-id>
image-name: ${{ steps.image_tag.outputs.image_tag }}
If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.
dbt code deploys
If you have a dbt project that you want to use with the deploy action for a GitHub action, you can choose to use either the dbt deploy on its own, or you can create GitHub actions that combine the infer
action with a dbt deploy.
Setup
- Single branch
- Multiple branch
- Combine dbt and Astro project
Prerequisites
- The root folder name for the directory that contains your dbt project.
Implementation
To automate code deploys to a single Deployment using GitHub Actions for a dbt project, complete the following setup in a Git-based repository that hosts an Astro project:
-
Set the following as a GitHub secret:
ASTRO_API_TOKEN
: The value for your Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy dbt code
on:
push:
branches:
- main
env:
## Sets Deployment API credentials as environment variables
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <deployment id>
deploy-type: dbt
root-folder: <dbt-root-folder>
The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.
Configuration requirements
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have at least one API token with access to both of your Deployments.
- The root folder name for the directory that contains your dbt project.
Implementation
-
Set the following as GitHub secrets:
PROD_ASTRO_API_TOKEN
: The value for your production Workspace or Organization API token.DEV_ASTRO_API_TOKEN
: The value for your development Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy dbt code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API credential as an environment variable
ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <dev-deployment-id>
deploy-type: dbt
root-folder: <dbt-root-folder>
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets Prod Deployment API credential as an environment variable
ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <prod-deployment-id>
deploy-type: dbt
root-folder: <dbt-root-folder>
In addition to configuring the deploy action to deploy just your dbt changes, you can also configure the action to do a dbt deploy
and an infer
deploy.
Prerequisites
- The root folder name for the directory that contct.ains your dbt project
Implementation
To automate code deploys to a single Deployment using GitHub Actions for a dbt project, complete the following setup in a Git-based repository that hosts both an Astro project and a dbt project:
-
Set the following as a GitHub secret:
ASTRO_API_TOKEN
: The value for your Workspace or Organization API token.
-
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy dbt and Astro project code
on:
push:
branches:
- main
env:
## Sets Deployment API credentials as environment variables
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: DBT Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <deployment id>
deploy-type: dbt
root-folder: <dbt-root-folder>
- name: DAGs/Image Deploy to Astro
uses: astronomer/deploy-action@v0.7.1
with:
deployment-id: <deployment id>
root-folder: <astro-project-root-folder>
parse: true