Configure a custom registry for Deployment images

Astro Private Cloud (APC) includes access to a Docker image registry that is installed by the Astronomer Helm chart, for data plane and unified mode. Every time a user deploys to APC, a Docker image is generated and pushed to this registry. Depending on your deploy method, these Docker images can include OS and Python dependencies, Dag code, and the Airflow service. Using the Astronomer-provided container image registry shipped with APC is recommended when you’re getting started and your team is comfortable deploying code. However, the Astronomer registry might not meet your organization’s security requirements. If your organization can’t support the Astronomer default internal registry, you can configure a custom container image registry. This option is best suited for organizations who require additional control for security and governance reasons. Using a custom registry provides your organization with the opportunity to scan images for CVEs, malicious code, and unapproved Python and OS-level packages contained in Docker images.

A custom registry can still connect to public networks or internet. Therefore, this procedure is different if you’re installing Astronomer in an air gapped environment. If you need to create a custom registry for a system that can’t connect to the public networks or internet, follow the configuration steps in the Air gapped tabs.

These instructions don’t apply to images hosted on Amazon Elastic Container Registry (ECR). Credentials for ECR have a limited lifespan and are unsuitable for using on APC. To use AWS ECR to serve images for Astro Private Cloud, you must grant permissions for the following actions to the Kubernetes Nodes IAM Role.

  "ecr:GetDownloadUrlForLayer",
  "ecr:BatchGetImage"

Prerequisites

Helm.
kubectl.
Astro CLI version 1.3.0+.
A custom container image registry.
A process for building and pushing your Astro projects as images to your custom registry.

Setup

Standard
Air gapped

Create a secret for the container repository credentials in your Astronomer namespace:

kubectl -n <astronomer-platform-namespace> create secret docker-registry <name-of-secret> \
  --docker-server=<your-registry-server> \
  --docker-username=<your-name> \
  --docker-password=<your-password> \
  --docker-email=<your-email>

To have Astro Private Cloud sync the registry credentials to all Deployment namespaces, add the following annotation:

kubectl -n <astronomer-platform-namespace> annotate secret <name-of-secret> "astronomer.io/commander-sync"="platform=astronomer"

To use different registries for each Deployment, create the same secret in each Deployment namespace instead of your Astronomer namespace. Make sure to specify different custom registries using --docker-server. If you don’t need to synch your secrets between Deployments, you don’t need to add the secret annotation.

In the APC UI, go to your Clusters page and select your cluster.

In the cluster details page, click Edit. In the Cluster Deployments Configuration YAML editor, click Find and search for the feature you want to override. Then add the following override in the appropriate YAML field:

astronomer:
  houston:
    config:
      deployments:
        deploymentImagesRegistry:
          updateDeploymentImageEndpoint:
            enabled: true
      registry:
        protectedCustomRegistry:
          enabled: true
          updateRegistry:
            enabled: true
            host: <airflow-image-repo>
            secretName: <name-of-secret>

For details on using the UI for configuration, see Override base configuration.

To use different registries for each Deployment, omit the astronomer.houston.config.deployments.registry.protectedCustomRegistry.updateRegistry.host setting.If you do set the host, use the format <registry>/<repo-name>/<subdirectory>, where <repo-name>/<subdirectory> includes the repository name and optionally a subdirectory within that repository.

Save and apply your changes in the UI.
For any existing Deployments, run the following command to sync the registry credentials.
kubectl create job -n <astronomer-platform-namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization
If you’re using different registries for each Deployment, skip this step.

Air gapped

Create a secret for the container repository credentials in your Astronomer namespace:
kubectl -n <your-namespace> create secret docker-registry <name-of-secret> \ --docker-server=<your-registry-server> \ --docker-username=<your-name> \ --docker-password=<your-password> \ --docker-email=<your-email>
To have Astro Private Cloud sync the registry credentials to all Deployment namespaces, add the following annotation:
kubectl -n <astronomer-platform-namespace> annotate secret <name-of-secret> "astronomer.io/commander-sync"="platform=astronomer"
To use different registries for each Deployment, create the same secret in each Deployment namespace instead of your Astronomer namespace. Make sure to specify different custom registries using --docker-server. You don’t need to add the annotation if you’re not syncing secrets between Deployments.
In the APC UI, go to your Clusters page and select your cluster.

astronomer:
  houston:
    config:
      deployments:
        deploymentImagesRegistry:
          updateDeploymentImageEndpoint:
            enabled: true
        helm:
          airflow:
            defaultAirflowRepository: <airflow-image-repo>
            images:
              airflow:
                repository: <airflow-image-repo>
      registry:
        protectedCustomRegistry:
          enabled: true
          baseRegistry:
            enabled: true
            host: <airflow-image-repo>
            secretName: <name-of-secret-containing-image-repo-creds>
          updateRegistry:
            enabled: true
            host: <airflow-image-repo>
            secretName: <name-of-secret-containing-image-repo-creds>

For details on using the UI for configuration, see Override base configuration.

Save and apply your changes in the UI.
For any existing Deployments, run the following command to sync the registry credentials. If you’re using different registries for each Deployment, you can skip this step.
kubectl create job -n <astronomer-platform-namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization

Push code to a custom registry

You can use the Astro CLI to build and push images to your custom registry. Based on the Helm configurations in your Astronomer cluster, the Astro CLI automatically detects your custom image registry and pushes your image to it. It then calls the APC API to update your Deployment to pull the new image from the registry. After you configure your custom registry, open your Astro project and run:

astro deploy

Alternatively, you can run a GraphQL mutation to update the image in your Deployment after manually pushing the image to the custom registry. This can be useful for automating code deploys using CI/CD. You can run a GraphQL mutation to update the image in your Deployment after manually pushing the image to a custom registry. This can be useful for automating code deploys using CI/CD. At a minimum, your mutation has to include the following:

mutation updateDeploymentImage {
        updateDeploymentImage(
                releaseName: "<deployment-release-name>", # for example "analytics-dev"
                image: "<host>/<image-name>:<tag>",  # for example docker.io/cmart123/ap-airflow:test4
                runtimeVersion: "<runtime-version-number>" # for example "5.0.6"
        )
        {
                id
        }
}

Alternatively, you can run this same mutation using cURL:

curl 'https://houston.BASEDOMAIN/v1' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: <your-token>' \
  --data-binary '{"query":"mutation updateDeploymentImage {updateDeploymentImage(releaseName: \"<deployment-release-name>\", image: \"<host>/<image-name>:<tag>\",runtimeVersion: \"<runtime-version-number>\"){id}}"}'

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Configure a custom registry for Deployment images

Prerequisites

Setup

Air gapped

Push code to a custom registry

​Prerequisites

​Setup

​Air gapped

​Push code to a custom registry

Prerequisites

Setup

Air gapped

Push code to a custom registry