GitHub Actions templates for deploying to Astro from private networks

The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See Deploy code with the Astro GitHub integration for setup steps.

If you don’t have access to the Astronomer deploy action because you can’t access the public internet from your GitHub repository, use one of the following private network templates to deploy to Astro. Read the following sections to choose the right template for your use case. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation. If you want to deploy a prebuilt image from an artifact registry without rebuilding, use the prebuilt image implementation. You can configure your CI/CD pipelines to deploy a full project image or your dags directory. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

If you use a self-hosted runner to execute jobs from GitHub Actions, the Astro CLI’s config.yaml file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:

Add ASTRO_API_TOKEN to your repository and include a check in your GitHub workflow to verify that it exists.
Use Deployment API tokens, which are scoped only to one Deployment, instead of Workspace or Organization API tokens.
Specify deployment-id or deployment-name in your action. For example, astro deploy <deployment-id> or astro deploy -n <deployment-name>.
Add the command astro logout at the end of your workflow to ensure that your authentication token is cleared from the config.yaml file.

Prerequisites

An Astro project hosted in a GitHub repository.
An Astro Deployment.
A Deployment API token, Workspace API token, or Organization API token.
Access to GitHub Actions.

Setup

Single branch
Multiple branch
Custom Image
Prebuilt image

To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:

Set the following as GitHub secrets:

ASTRO_API_TOKEN: The value for your Workspace or Organization API token.

In your project repository, create a new YAML file in .github/workflows that includes the following configuration. When you make a commit to a specified branch, this workflow sets your Deployment API credentials as environment variables, installs the latest version of the Astro CLI, checks to see if your dags folder has changes, and then either completes a full code deploy or a dag-only code deploy.

name: Astronomer CI - Deploy code

on:
  push:
    branches:
      - main

env:
  ## Sets Deployment API credentials as environment variables
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  build:
    runs-on: ubuntu-latest # add the appropriate image
    steps:
        # Install the Astro CLI (current version)
        - name: checkout repo
          uses: actions/checkout@v3
          with:
            fetch-depth: 2
            clean: false
        - name: Install the CLI
          run: curl -sSL install.astronomer.io | sudo bash -s
        # Determine if only dag files have changes
        - name: Deploy to Astronomer
          run: |
            files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
            dags_only=1
            for file in $files; do
            if [[ $file != dags/* ]]; then
                echo "$file is not a dag, triggering a full image build"
                dags_only=0
                break
            fi
            done
            ### If only dags changed deploy only the dags in your 'dags' folder to your Deployment
            if [ $dags_only == 1 ]
            then
                astro deploy --dags
            fi
            ### If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and dag changes
            if [ $dags_only == 0 ]
            then
                astro deploy
            fi

The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions to push a full image deploy to Astro. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

Prerequisites (Multiple branch)

You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
You have respective dev and prod Deployments on Astro where you deploy your GitHub branches to.
You have at least one API token with access to both of your Deployments.

Setup (Multiple branch)

Set the following as GitHub secrets:

PROD_ASTRO_API_TOKEN: The value for your production Workspace or Organization API token.
DEV_ASTRO_API_TOKEN: The value for your Workspace or Organization API token.

In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

name: Astronomer CI - Deploy code (Multiple Branches)

on:
  push:
    branches: [dev]
  pull_request:
    types:
      - closed
    branches: [main]

jobs:
  dev-push:
    if: github.ref == 'refs/heads/dev'
    env:
      ## Sets DEV Deployment API token credential as an environment variable
      ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
    runs-on: ubuntu-latest
    steps:
    - name: checkout repo
      uses: actions/checkout@v3
    - name: Deploy to Astro
      run: |
        curl -sSL install.astronomer.io | sudo bash -s
        astro deploy <your-dev-deployment-id>
  prod-push:
    if: github.event.action == 'closed' && github.event.pull_request.merged == true
    env:
      ## Sets PROD Deployment API token credential as an environment variable
      ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
    runs-on: ubuntu-latest
    steps:
    - name: checkout repo
      uses: actions/checkout@v3
    - name: Deploy to Astro
      run: |
        curl -sSL install.astronomer.io | sudo bash -s
        astro deploy <your-prod-deployment-id>

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker’s build-push-action. This template always pushes your entire project as an image to Astro.

Prerequisites (Custom Image)

An Astro project that requires additional build-time arguments to build the Runtime image.

Setup (Custom Image)

Set the following as GitHub secrets:

ASTRO_API_TOKEN: The value for your Workspace or Organization API token.

In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

name: Astronomer CI - Additional build-time args

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
    steps:
    - name: Check out the repo
      uses: actions/checkout@v3
    - name: Create image tag
      id: image_tag
      run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
    - name: Build image
      uses: docker/build-push-action@v4
      with:
        tags: ${{ steps.image_tag.outputs.image_tag }}
        load: true
        # Define your custom image's build arguments, contexts, and connections here using
        # the available GitHub Action settings:
        # https://github.com/docker/build-push-action#customizing .
        # This example uses `build-args` , but your use case might require configuring
        # different values.
        build-args: |
          <your-build-arguments>
    - name: Deploy to Astro
      run: |
        curl -sSL install.astronomer.io | sudo bash -s
        astro deploy <your-deployment-id> --image-name ${{ steps.image_tag.outputs.image_tag }}

For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:

name: Astronomer CI - Custom base image

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
    steps:
    - name: Check out the repo
      uses: actions/checkout@v3
    - name: Create image tag
      id: image_tag
      run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
    - name: Create SSH Socket
      uses: webfactory/ssh-agent@v0.5.4
      with:
        # GITHUB_SSH_KEY must be defined as a GitHub secret.
        ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
    - name: (Optional) Test SSH Connection - Should print hello message.
      run: (ssh git@github.com) || true
    - name: Build image
      uses: docker/build-push-action@v2
      with:
        tags: ${{ steps.image_tag.outputs.image_tag }}
        load: true
        ssh: |
          github=${{ env.SSH_AUTH_SOCK }}
    - name: Deploy to Astro
      run: |
        curl -sSL install.astronomer.io | sudo bash -s
        astro deploy <your-deployment-id> --image-name ${{ steps.image_tag.outputs.image_tag }}

If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.

If your organization builds Astro Runtime images in a separate build pipeline and stores them in an artifact registry, use this template to pull and deploy the prebuilt image without rebuilding it. This template skips the build step entirely and deploys the image directly from your registry to Astro.

Prerequisites (Prebuilt image)

A prebuilt Docker image based on Astro Runtime, stored in an artifact registry such as Google Artifact Registry, Amazon ECR, Azure Container Registry, or Docker Hub.
Registry credentials with pull access to the image.

Setup (Prebuilt image)

Set the following as GitHub secrets:

ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
Registry credentials for your artifact registry. See the following tabs for registry-specific secrets.

In your project repository, create a new YAML file in .github/workflows that includes the following configuration for your registry:

Google Artifact Registry
Amazon ECR
Azure Container Registry
Docker Hub

name: Astronomer CI - Deploy prebuilt image from Google Artifact Registry

on:
  push:
    branches:
      - main

env:
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
  REGISTRY: <your-region>-docker.pkg.dev/<your-project-id>/<your-repository>
  IMAGE_NAME: <your-image-name>
  IMAGE_TAG: <your-image-tag>

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Authenticate to Google Cloud
      uses: google-github-actions/auth@v2
      with:
        credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
    - name: Configure Docker for Google Artifact Registry
      run: gcloud auth configure-docker <your-region>-docker.pkg.dev --quiet
    - name: Pull image from registry
      run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
    - name: Install the Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy to Astro
      run: |
        astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}

Set GCP_SERVICE_ACCOUNT_KEY as a GitHub secret containing the JSON key for a Google Cloud service account with read access to the registry.

name: Astronomer CI - Deploy prebuilt image from Amazon ECR

on:
  push:
    branches:
      - main

env:
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
  REGISTRY: <your-account-id>.dkr.ecr.<your-region>.amazonaws.com
  IMAGE_NAME: <your-image-name>
  IMAGE_TAG: <your-image-tag>

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: <your-region>
    - name: Authenticate to Amazon ECR
      uses: aws-actions/amazon-ecr-login@v2
    - name: Pull image from registry
      run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
    - name: Install the Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy to Astro
      run: |
        astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}

Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as GitHub secrets with permissions to pull from the ECR repository.

name: Astronomer CI - Deploy prebuilt image from Azure Container Registry

on:
  push:
    branches:
      - main

env:
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
  REGISTRY: <your-registry-name>.azurecr.io
  IMAGE_NAME: <your-image-name>
  IMAGE_TAG: <your-image-tag>

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Authenticate to Azure Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ secrets.ACR_USERNAME }}
        password: ${{ secrets.ACR_PASSWORD }}
    - name: Pull image from registry
      run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
    - name: Install the Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy to Astro
      run: |
        astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}

Set ACR_USERNAME and ACR_PASSWORD as GitHub secrets. You can use a service principal or an admin account with pull access to the registry.

name: Astronomer CI - Deploy prebuilt image from Docker Hub

on:
  push:
    branches:
      - main

env:
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
  IMAGE_NAME: <your-dockerhub-org>/<your-image-name>
  IMAGE_TAG: <your-image-tag>

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Authenticate to Docker Hub
      uses: docker/login-action@v3
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}
    - name: Pull image from registry
      run: docker pull ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
    - name: Install the Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy to Astro
      run: |
        astro deploy <your-deployment-id> --image-name ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}

Set DOCKERHUB_USERNAME and DOCKERHUB_TOKEN as GitHub secrets with pull access to the repository.

These workflows authenticate to the registry, pull the prebuilt image, and deploy it to Astro using --image-name. No build step occurs. The image must be based on Astro Runtime.To learn more about deploying prebuilt images, see Deploy a prebuilt Docker image.

Overview

Get started

AI features

Build and deploy

Run and operate

Observe and monitor

Administration

Best practices

Reference

Release notes

Airflow 3

GitHub Actions templates for deploying to Astro from private networks

Prerequisites

Setup

Prerequisites (Multiple branch)

Setup (Multiple branch)

Prerequisites (Custom Image)

Setup (Custom Image)

Prerequisites (Prebuilt image)

Setup (Prebuilt image)

​Prerequisites

​Setup

​Prerequisites (Multiple branch)

​Setup (Multiple branch)

​Prerequisites (Custom Image)

​Setup (Custom Image)

​Prerequisites (Prebuilt image)

​Setup (Prebuilt image)

Prerequisites

Setup

Prerequisites (Multiple branch)

Setup (Multiple branch)

Prerequisites (Custom Image)

Setup (Custom Image)

Prerequisites (Prebuilt image)

Setup (Prebuilt image)