Version:

v0.10.0

Documentation

CI/CD


With Astronomer's support for Service accounts, you're able to deploy to an Apache Airflow deployment via a Continuous Integration/Continuous Delivery (CI/CD) tool of your choice.

This guide will walk you through configuring your CI/CD pipeline with either Astronomer Cloud or Astronomer Enterprise.

For background and best practices on CI/CD, we recommend reading "An Introduction to CI/CD Best Practices" from DigitalOcean.

Pre-Requisites

Before we get started, make sure you:

  • Have access to a running Airflow Deployment on either Astronomer Cloud or Enterprise
  • Installed the Astronomer CLI
  • Are familiar with your CI/CD tool of choice

Create a Service Account

In order to authenticate your CI/CD pipeline to the private Docker registry, you'll need to create a service account. Once created, you'll be able to delete this service account at any time either via the CLI or via the Astronomer UI. In both cases, this will generate an API key that will be used for the CI/CD process.

Note that you're able to create Service Accounts at the:

  • Workspace Level
  • Airflow Deployment Level

Creating a Service Account at the Workspace level allows you to deploy to multiple Airflow deployments with one code push, while creating them at the Deployment level ensures that your CI/CD pipeline only deploys to one particular deployment on Astronomer.

Read below for guidelines on how to create a service account via the CLI and via the Astronomer UI.

Create a Service Account via the CLI

Deployment Level Service Account

To create a Deployment Level Service account via the CLI, first run:

$ astro deployment list

This will output the list of running Airflow deployments you have access to, and their corresponding UUID.

With that UUID, run:

$ astro service-account create -d [DEPLOYMENTUUID] --label [SERVICEACCOUNTLABEL]

Workspace Level Service Account

To create a Workspace Level Service account via the CLI, first run:

$ astro workspace list

This will output the list of running Astronomer Workspaces you have access to, and their corresponding UUID.

With that UUID, run:

$ astro service-account create -w [WORKSPACEUUID] -l [SERVICEACCOUNTLABEL]

Create a Service Account via the Astronomer UI

If you prefer to provision a Service Account through the Astronomer UI, start by logging into Astronomer.

Navigate to your Deployment's "Configure" Page

From the Astronomer UI, navigate to: Deployment > Service Accounts

New Service Account

Give your Service Account a Name and Category

Upon creating a Service Account, give it a Name and Category.

Name Service Account

Copy the API Key

Lastly, grab the API Key generated by creating the Service Account.

Note: This API key will only be visible during the session.

Service Account

Authenticate to Docker

Once you've created a service account, you will want to store the generated API key in an environment variable, or your secret management tool of choice.

The first step of this pipeline is to authenticate against the registry:

docker login registry.$${BASE_DOMAIN} -u _ -p $${API_KEY_SECRET}

In this example:

  • BASEDOMAIN = gcp0001.us-east4.astronomer.io (for Astronomer Cloud users) or your very own Basedomain for Enterprise
  • API_KEY_SECRET = The API Key that you got from the CLI or the UI and stored in your secret manager

Building and Pushing an Image

Once you are authenticated you can build, tag and push your Airflow image to the private registry, where a webhook will trigger an update to your Airflow deployment on the platform.

Registry Address

Registry Address tells Docker where to push images to. In this case it will either be:

  • Astronomer Enterprise: Your private registry located at registry.${BASE_DOMAIN}.

  • Astronomer Cloud: registry.gcp0001.us-east4.astronomer.io

Release Name

Release Name refers to the release name of your Airflow Deployment. It will follow the pattern of spaceyword-spaceyword-4digits (e.g. infrared-photon-7780).

Tag Name

Tag Name allows you to track all Airflow deployments made for that cluster over time. While the tag name can be whatever you want, we recommend denoting the source and the build number in the name.

In the below example we use the prefix ci- and the ENV ${DRONE_BUILD_NUMBER}. This guarentees that we always know which CI/CD build triggered the build and push.

Example:

docker build -t registry.${BASE_DOMAIN}/${RELEASE_NAME}/airflow:ci-${DRONE_BUILD_NUMBER} .

If you would like to see a more complete working example please visit our full example using Drone-CI.

Configure Your CI/CD Pipeline

Depending on your CI/CD tool, configuration will be slightly different. This section will focus on outlining what needs to be accomplished, not the specifics of how.

At its core, your CI/CD pipeline will be authenticating to the private registry installed with the platform, then building, tagging and pushing an image to that registry.

Note: The base image is based on the version of Astronomer you are currently running.

DroneCI

pipeline:
  build:
    image: astronomerio/ap-build:0.0.7
    commands:
      - docker build -t registry.gcp0001.us-east4.astronomer.io/infrared-photon-7780/airflow:ci-${DRONE_BUILD_NUMBER} .
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: push
      branch: [ master, release-* ]

  push:
    image: astronomerio/ap-build:0.0.7
    commands:
      - echo $${DOCKER_PASSWORD_TEST}
      - docker login registry.gcp0001.us-east4.astronomer.io -u _ -p $${DOCKER_PASSWORD_TEST}
      - docker push registry.gcp0001.us-east4.astronomer.io/infrared-photon-7780/airflow:ci-${DRONE_BUILD_NUMBER}
    secrets: [ docker_password_test ]
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: push
      branch: [ master, release-* ]

CircleCI

# Python CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-python/ for more details
#
version: 2
jobs:
  build:
    machine: true
    steps:
      - checkout
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "requirements.txt" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-
      - run:
          name: run linter
          command: |
            pycodestyle .

      - save_cache:
          paths:
            - ./venv
          key: v1-dependencies-{{ checksum "requirements.txt" }}
  deploy:
    docker:
      - image:  astronomerio/ap-build:0.0.7
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Push to Docker Hub
          command: |
            TAG=0.1.$CIRCLE_BUILD_NUM
            docker build -t registry.gcp0001.us-east4.astronomer.io/empty-isotope-6604/airflow:ci-$TAG .
            docker login registry.gcp0001.us-east4.astronomer.io -u _ -p $DOCKER_KEY
            docker push registry.gcp0001.us-east4.astronomer.io/empty-isotope-6604/airflow:ci-$TAG

workflows:
  version: 2
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only:
                - master

Jenkins Script

pipeline {
 agent any
   stages {
     stage('Deploy to astronomer') {
       when { branch 'master' }
       steps {
         script {
           sh 'docker build -t registry.gcp0001.us-east4.astronomer.io/foo-bar-8077/airflow:ci-${BUILD_NUMBER} .'
           sh 'docker login registry.gcp0001.us-east4.astronomer.io -u _ -p ${ASTRO_KEY}'
           sh 'docker push registry.gcp0001.us-east4.astronomer.io/foo-bar-8077/airflow:ci-${BUILD_NUMBER}'
         }
       }
     }
   }
 post {
   always {
     cleanWs()
   }
 }
}

Bitbucket

If you are using Bitbucket, this script should work (courtesy of our friends at Das42)

image: astronomerio/ap-build:0.0.7

pipelines:
  branches:
    master:
      - step:
          name: Deploy to production
          deployment: production
          script:
            - echo ${ASTRONOMER_PASSWORD}
            - docker login registry.gcp0001.us-east4.astronomer.io -u _ -p ${ASTRONOMER_PASSWORD}
            - docker push registry.gcp0001.us-east4.astronomer.io/infrared-photon-7780/airflow:ci-${BITBUCKET_BUILD_NUMBER}
          services:
            - docker
          caches:
            - docker

Gitlab

astro_deploy:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Building container.."
    - docker build -t registry.gcp0001.us-east4.astronomer.io/cometary-crater-7965/airflow:CI-$CI_PIPELINE_IID .
    - docker login rregistry.gcp0001.us-east4.astronomer.io -u _ -p $${SERVICE_ACCOUNT_SECRET}
    - docker push registry.gcp0001.us-east4.astronomer.io/cometary-crater-7965/airflow:CI-$CI_PIPELINE_IID
  only:
    - master

Video Tutorial

Check out this video for a full walkthrough of this process: