Configure CI/CD on Astronomer Software

Users can push code and deploy to an Airflow Deployment on Astronomer using a Continuous Integration/Continuous Delivery (CI/CD) tool.

There are many benefits to deploying DAGs and other changes to Airflow via a CI/CD workflow. Specifically, you can:

• Deploy new and updated DAGs in a way that streamlines the development process amongst team members.
• Decrease the maintenance cost of integrating changes, allowing your team to quickly respond in case of an error or failure.
• Enforce continuous, automating testing, which increases code quality and protects your DAGs in production.

This guide will walk you through configuring a CI/CD pipeline on Astronomer.

Example CI/CD workflow

Consider an Astro project hosted on GitHub and deployed to Astronomer. In this scenario, a set of dev and main branches of an Astro project are hosted on a single GitHub repository, and dev and prod Airflow Deployments are hosted on an Astronomer Workspace.

Using CI/CD, you can automatically deploy DAGs to your Airflow Deployment on Astronomer by pushing or merging code to a corresponding branch in GitHub. The general setup would look something like this:

1. Create two Airflow Deployments within your Astronomer Workspace, one for dev and one for prod.
2. Create a repository in GitHub that hosts project code for all Airflow Deployments within your Astronomer Workspace.
3. In your GitHub code repository, create a dev branch off of your main branch.
4. Configure your CI/CD tool to deploy to your dev Airflow Deployment whenever you push to your dev branch, and to deploy to your prod Airflow Deployment whenever you merge your dev branch into main.

That would look something like this:

CI/CD on Astronomer

Automating the deploy process to Astronomer starts with creating a Service Account, which will assume a user role and some set of permissions to your Workspace or Deployment.

From there, you’ll write a script that allows your service account to do the following:

• Build and tag a Docker Image
• Authenticate to a Docker Registry
• Push your image to that Docker Registry

From there, a webhook triggers an update to your Airflow Deployment using the CI/CD tool of your choice. The Astro CLI completes an authentication and push for every manual $ astro deploy command.

The rest of this guide describes how to create a service account and what your CI/CD script should look like based on the tool you’re using.

Prerequisites

Before completing this setup, make sure you:

• Have access to a running Airflow Deployment on Astronomer.
• Installed the Astro CLI.
• Are familiar with your CI/CD tool of choice.

Step 1: Create a service account

In order to authenticate your CI/CD pipeline to Astronomer’s private Docker registry, you’ll need to create a service account and grant it an appropriate set of permissions. You can do so via the Software UI or CLI. Once created, you can always delete this service account at any time. In both cases, creating a service account will generate an API key that will be used for the CI/CD process.

Note that you’re able to create Service Accounts at the:

• Workspace Level
• Airflow Deployment Level

Creating a service account at the Workspace level allows you to deploy to multiple Airflow deployments with one code push, while creating them at the Deployment level ensures that your CI/CD pipeline only deploys to one particular deployment on Astronomer.

Read below for guidelines on how to create a service account via the CLI and via the Software UI.

Create a service account using the CLI

Deployment level service account

To create a Deployment Level Service account via the CLI, first run:

$
astro deployment list

This will output the list of running Airflow deployments you have access to, and their corresponding UUID.

With that UUID, run:

$
astro deployment service-account create -d <deployment-id> --label <service-account-label> --role <deployment-role>

Workspace level service account

To create a Workspace Level Service account via the CLI, first run:

$
astro workspace list

This will output the list of running Astronomer Workspaces you have access to, and their corresponding UUID.

With that UUID, run:

$
astro workspace service-account create -w <workspace-id> --label <service-account-label> --role <workspace-role>

Create a service account using the Software UI

If you prefer to provision a service account through the Software UI, start by logging into Astronomer.

Go to your Deployment’s “Configure” page

From the Software UI, navigate to: Deployment > Service Accounts

Configure your service account

Upon creating a Service Account, make sure to:

• Give it a Name
• Give it a Category (optional)
• Grant it a User Role

Copy the API key

Once you’ve created your new Service Account, grab the API key that was immediately generated. Depending on your use case, you might want to store this key in an Environment Variable or secret management tool of choice.

Step 2: Authenticate and push to Docker

The first step of this pipeline will authenticate against the Docker registry that stores an individual Docker image for every code push or configuration change:

$
docker login registry.${BASE_DOMAIN} -u _ -p $${API_KEY_SECRET}

In this example:

• BASE_DOMAIN = The domain at which your Software instance is running
• API_KEY_SECRET = The API key that you got from the CLI or the UI and stored in your secret manager

Building and pushing an image

Once you are authenticated you can build, tag and push your Airflow image to the private registry, where a webhook will trigger an update to your Airflow Deployment on the platform.

Registry address

Registry Address tells Docker where to push images to. On Astronomer Software, your private registry is located at registry.${BASE_DOMAIN}.

Release name

Release Name refers to the release name of your Airflow Deployment. It will follow the pattern of spaceyword-spaceyword-4digits (e.g. infrared-photon-7780).

Tag name

Tag Name: Each deploy to Astronomer generates a Docker image with a corresponding tag. If you deploy via the CLI, the tag will by default read deploy-n, with n representing the number of deploys made to that Airflow Deployment. If you’re using CI/CD, you get to customize this tag. We typically recommend specifying the source and the build number in the name.

In the example below, we use the prefix ci- and the ENV ${DRONE_BUILD_NUMBER}. This guarantees that we always know which CI/CD build triggered the build and push.

Example:

$
docker build -t registry.${BASE_DOMAIN}/${RELEASE_NAME}/airflow:ci-${DRONE_BUILD_NUMBER} .

If you would like to see a more complete working example please visit our full example using Drone-CI.

Run unit tests

For CI/CD pipelines that push code to a production Deployment, Astronomer recommends adding a unit test after the image build step to ensure that you don’t push a Docker image with breaking changes. To run a basic unit test, add a step in your CI/CD pipeline that executes docker run and then runs pytest tests in a container that is based on your newly built image before it’s pushed to your registry.

For example, you can add the following command as a step in a Drone pipeline:

$
docker run --rm registry.${BASE_DOMAIN}/${RELEASE_NAME}/airflow:ci-${DRONE_BUILD_NUMBER} /bin/bash -c "pytest tests"

Step 3: Configure your CI/CD pipeline

Depending on your CI/CD tool, configuration will be slightly different. This section will focus on outlining what needs to be accomplished, not the specifics of how.

At its core, your CI/CD pipeline will first authenticate to Astronomer’s private registry and then build, tag and push your Docker image to that registry.

Example implementation

The following setup is an example implementation of CI/CD using GitHub Actions. These steps cover both the implementation and the workflow necessary to create a fully functional CI/CD pipeline.

1. Create a GitHub repository for an Astro project. Ensure your repo has a development branch and a main branch. In this example, the branches are named dev and main.

2. Create two Deployment-level service accounts: One for your Dev Deployment and one for your Production Deployment.

3. Follow instructions in GitHub documentation to add your service accounts as secrets to your repository. In the example below, these secrets are named SERVICE_ACCOUNT_KEY and SERVICE_ACCOUNT_KEY_DEV.

4. Go to the Actions tab of your GitHub repo and create a new action with a main.yml file. To achieve the recommended workflow, use the following action:

   1
   name: Astronomer CI - Deploy code
   2
   on:
   3
     push:
   4
       branches: [dev]
   5
     pull_request:
   6
       types:
   7
         - closed
   8
       branches: [main]
   9
   jobs:
   10
     dev-push:
   11
       if: github.ref == 'refs/heads/dev'
   12
       runs-on: ubuntu-latest
   13
       steps:
   14
       - name: Check out the repo
   15
         uses: actions/checkout@v3
   16
       - name: Log in to registry
   17
         uses: docker/login-action@v1
   18
         with:
   19
           registry: registry.${BASE_DOMAIN}
   20
           username: _
   21
           password: ${{ secrets.SERVICE_ACCOUNT_KEY_DEV }}
   22
       - name: Build image
   23
         run: docker build -t registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }} .
   24
       - name: Run tests
   25
         run: docker run --rm registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }} /bin/bash -c "pytest tests"
   26
       - name: Push image
   27
         run: docker push registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }}
   28
     prod-push:
   29
       if: github.event.action == 'closed' && github.event.pull_request.merged == true
   30
       runs-on: ubuntu-latest
   31
       steps:
   32
       - name: Check out the repo
   33
         uses: actions/checkout@v3
   34
       - name: Log in to registry
   35
         uses: docker/login-action@v1
   36
         with:
   37
           registry: registry.${BASE_DOMAIN}
   38
           username: _
   39
           password: ${{ secrets.SERVICE_ACCOUNT_KEY }}
   40
       - name: Build image
   41
         run: docker build -t registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }} .
   42
       - name: Run tests
   43
         run: docker run --rm registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }} /bin/bash -c "pytest tests"
   44
       - name: Push image
   45
         run: docker push registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }}

   Ensure the branches match the names of the branches in your repository, and replace <dev-release-name> and <prod-release-name> with the respective release names of your development and production Airflow Deployments on Astronomer.

5. Test the GitHub Action by making a change on your dev branch and committing that change. This should update your development Airflow Deployment on Astronomer, which you can confirm in the Software UI. If that update was successful, try opening a pull request and then merging dev into main to update your production Airflow Deployment. If both updates were successful, you now have a functioning, scalable CI/CD pipeline that can automatically deploy code to multiple Airflow Deployments.

The following sections provide basic templates for configuring single CI/CD pipelines using popular CI/CD tools. Each template can be implemented to produce a simple CI/CD pipeline similar to the one above, but they can also be reconfigured to manage any number of branches or Deployments based on your needs.

DroneCI

1
pipeline:
2
  build:
3
    image: quay.io/astronomer/ap-build:latest
4
    commands:
5
      - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER} .
6
    volumes:
7
      - /var/run/docker.sock:/var/run/docker.sock
8
    when:
9
      event: push
10
      branch: [ master, release-* ]
11
12
  test:
13
    image: quay.io/astronomer/ap-build:latest
14
    commands:
15
      - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER} /bin/bash -c "pytest tests"
16
    volumes:
17
      - /var/run/docker.sock:/var/run/docker.sock
18
    when:
19
      event: push
20
      branch: [ master, release-* ]
21
22
  push:
23
    image: quay.io/astronomer/ap-build:latest
24
    commands:
25
      - echo $${SERVICE_ACCOUNT_KEY}
26
      - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
27
      - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER}
28
    secrets: [ SERVICE_ACCOUNT_KEY ]
29
    volumes:
30
      - /var/run/docker.sock:/var/run/docker.sock
31
    when:
32
      event: push
33
      branch: [ master, release-* ]

CircleCI

1
# Python CircleCI 2.0 configuration file
2
#
3
# Check https://circleci.com/docs/2.0/language-python/ for more details
4
#
5
version: 2
6
jobs:
7
  build:
8
    machine: ubuntu-2004:202008-01
9
    steps:
10
      - checkout
11
      - restore_cache:
12
          keys:
13
          - v1-dependencies-{{ checksum "requirements.txt" }}
14
          # fallback to using the latest cache if no exact match is found
15
          - v1-dependencies-
16
      - run:
17
          name: Install test deps
18
          command: |
19
            # Use a virtual env to encapsulate everything in one folder for
20
            # caching. And make sure it lives outside the checkout, so that any
21
            # style checkers don't run on all the installed modules
22
            python -m venv ~/.venv
23
            . ~/.venv/bin/activate
24
            pip install -r requirements.txt
25
      - save_cache:
26
          paths:
27
            - ~/.venv
28
          key: v1-dependencies-{{ checksum "requirements.txt" }}
29
      - run:
30
          name: run linter
31
          command: |
32
            . ~/.venv/bin/activate
33
            pycodestyle .
34
  deploy:
35
    docker:
36
      - image:  docker:18.09-git
37
    steps:
38
      - checkout
39
      - setup_remote_docker:
40
          docker_layer_caching: true
41
      - run:
42
          name: Push to Docker Hub
43
          command: |
44
            TAG=0.1.$CIRCLE_BUILD_NUM
45
            docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG .
46
            docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG /bin/bash -c "pytest tests"
47
            docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
48
            docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG
49
50
workflows:
51
  version: 2
52
  build-deploy:
53
    jobs:
54
      - build
55
      - deploy:
56
          requires:
57
            - build
58
          filters:
59
            branches:
60
              only:
61
                - master

Jenkins Script

1
pipeline {
2
    agent any
3
    stages {
4
        stage('Deploy to astronomer') {
5
            when { branch 'master' }
6
            steps {
7
                script {
8
                    sh 'docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER} .'
9
                    sh 'docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER} /bin/bash -c "pytest tests'
10
                    sh 'docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY'
11
                    sh 'docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER}'
12
                }
13
            }
14
        }
15
    }
16
    post {
17
        always {
18
            cleanWs()
19
        }
20
    }
21
}

Bitbucket

If you are using Bitbucket, this script should work (courtesy of our friends at Das42)

1
image: quay.io/astronomer/ap-build:latest
2
3
pipelines:
4
  branches:
5
    master:
6
      - step:
7
          name: Deploy to production
8
          deployment: production
9
          script:
10
            - echo ${SERVICE_ACCOUNT_KEY}
11
            - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER} .
12
            - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER} /bin/bash -c "pytest tests"
13
            - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
14
            - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER}
15
          services:
16
            - docker
17
          caches:
18
            - docker

Gitlab

1
astro_deploy:
2
  stage: deploy
3
  image: docker:latest
4
  services:
5
    - docker:dind
6
  script:
7
    - echo "Building container.."
8
    - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID .
9
    - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID /bin/bash -c "pytest tests"
10
    - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
11
    - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID
12
  only:
13
    - master

AWS Codebuild

1
version: 0.2
2
phases:
3
  install:
4
    runtime-versions:
5
      python: latest
6
7
  pre_build:
8
    commands:
9
      - echo Logging in to dockerhub ...
10
      - docker login "registry.$BASE_DOMAIN" -u _ -p "$API_KEY_SECRET"
11
      - export GIT_VERSION="$(git rev-parse --short HEAD)"
12
      - echo "GIT_VERSION = $GIT_VERSION"
13
      - pip install -r requirements.txt
14
15
  build:
16
    commands:
17
      - docker build -t "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION" .
18
      - docker run --rm "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION" /bin/bash -c "pytest tests"
19
      - docker push "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION"

GitHub Actions CI/CD

GitHub supports a growing set of native CI/CD features in “GitHub Actions”, including a “Publish Docker” action that works well with Astronomer.

To use GitHub Actions on Astronomer, create a new action in your repo at .github/workflows/main.yml with the following:

1
name: CI
2
3
on: [push]
4
5
jobs:
6
  build:
7
    runs-on: ubuntu-latest
8
    steps:
9
    - name: Check out the repo
10
      uses: actions/checkout@v3
11
    - name: Log in to registry
12
      uses: docker/login-action@v1
13
      with:
14
        registry: registry.$BASE_DOMAIN
15
        username: _
16
        password: ${{ secrets.SERVICE_ACCOUNT_KEY }}
17
    - name: Build image
18
      run: docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${{ github.sha }} .
19
    - name: Run tests
20
      run: docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${{ github.sha }} /bin/bash -c "pytest tests"
21
    - name: Push image
22
      run: docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${{ github.sha }}

Azure DevOps

In this example configuration, you can automatically deploy your Astro project from a GitHub repository using an Azure Devops pipeline connected to the GitHub repository.

To set up this workflow, make sure you have:

• A GitHub repository hosting your Astro project.
• An Azure DevOps account with permissions to create new pipelines.

1. Create a new file called astro-devops-cicd.yaml in your Astro project repository with the following configuration:

   1
   # Control which branches have CI triggers:
   2
   trigger:
   3
   - main
   4
   5
   # To trigger the build/deploy only after a PR has been merged:
   6
   pr: none
   7
   8
   # Optionally use Variable Groups & Azure Key Vault:
   9
   #variables:
   10
   #- group: Variable-Group
   11
   #- group: Key-Vault-Group
   12
   13
   stages:
   14
   - stage: build
   15
     jobs:
   16
     - job: run_build
   17
       pool:
   18
         vmImage: 'Ubuntu-latest'
   19
       steps:
   20
       - script: |
   21
           echo "Building container.."
   22
           docker build -t registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion) .
   23
           docker run --rm registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion) /bin/bash -c "pytest tests"
   24
           docker login registry.$(BASE-DOMAIN) -u _ -p $(SVC-ACCT-KEY)
   25
           docker push registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion)

2. Follow the steps in Azure documentation to link your GitHub Repo and Action to an Azure pipeline. When prompted for the source code for your pipeline, specify that you have an existing Azure Pipelines YAML file and provide the setup tool with the file path to your astro-devops-cicd.yaml file.

3. Finish and save your Azure pipeline setup.

4. In Azure, add environment variables for the following values:

   • BASE-DOMAIN: Your base domain for Astronomer
   • RELEASE-NAME: The release name for your Deployment
   • SVC-ACCT-KEY: The service account you created on Astronomer for CI/CD

   We recommend marking SVC-ACCT-KEY as secret.

Once you complete this setup, any merges on the main branch of your GitHub repo will trigger the pipeline and deploy your changes to Astronomer.