Run images from a private registry

By default, the KubernetesPodOperator expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly. Otherwise, if you are using any other private registry, you need to create a Kubernetes Secret containing credentials to the registry, then specify the Kubernetes Secret in your dag.

Prerequisites

An Astro project.
An Astro Deployment.
Access to a private Docker registry.

Setup

Create a Kubernetes Secret

To run Docker images from a private registry on Astro, a Kubernetes Secret that contains credentials to your registry must be created. Injecting this secret into your Deployment’s namespace will give your tasks access to Docker images within your private registry.

By default, the KubernetesPodOperator looks for publicly hosted images. However, you can pull images from a private registry.

Retrieve a config.json file that contains your Docker credentials by following the Docker documentation. The generated file looks similar to the following:

1 {
2     "auths": {
3         "https://index.docker.io/v1/": {
4             "auth": "c3R...zE2"
5         }
6     }
7 }

Submit a request to Astronomer support for creating a Kubernetes Secret to enable pulling images from private registries. Astronomer Support can provide you the necessary instructions on how to generate and securely send the credentials.

Specify the Kubernetes Secret in your dag

Astronomer adds the Kubernetes secret to your Deployment, Astronomer notifies you and provides you with the name of the secret.
After you receive the name of your Kubernetes secret from Astronomer, you can run images from your private registry by importing models from kubernetes.client and configuring image_pull_secrets in your KubernetesPodOperator instantiation:

1 from kubernetes.client import models as k8s
2 
3 KubernetesPodOperator(
4     namespace=namespace,
5     image_pull_secrets=[k8s.V1LocalObjectReference("<your-secret-name>")],
6     image="<your-docker-image>",
7     cmds=["<commands-for-image>"],
8     arguments=["<arguments-for-image>"],
9     labels={"<pod-label>": "<label-name>"},
10     name="<pod-name>",
11     task_id="<task-name>",
12     get_logs=True,
13     in_cluster=True,
14 )