Run images from Google Artifact Registry

Passwordless setup is available only on Astro dedicated clusters. For Astro standard clusters, follow the steps in Private Registry to create a Kubernetes secret containing your registry credentials.

By default, the KubernetesPodOperator expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly.

Prerequisites

An Astro project.
An Astro Deployment.
Access to your Google Artifact Registry repository.

Setup

If your container image is hosted in Google Artifact Registry repository, add a permissions policy to the repository to allow the KubernetesPodOperator to pull the Docker image. You don’t need to create a Kubernetes secret or specify the Kubernetes secret in your dag. Docker images hosted in Google Artifact Registry repositories can be pulled only to Deployments hosted on GCP clusters.

Request the Compute Engine default service account ID

Contact Astronomer support to request the Compute Engine default service account ID for your cluster.

Add Google Artifact Registry repository permissions

Log in to Google Artifact Registry.
Click the checkbox next to the repository that you want to use.
In the Properties pane that appears, click ADD PRINCIPAL in the PERMISSIONS tab.
In the Add Principals text box, paste the Compute Engine default service account ID that was provided to you by Astronomer Support.
In the Assign Roles selector, search for Artifact Registry Reader and select the role that appears.
Click Save to grant read access for the registry to Astro.

Set up the KubernetesPodOperator

The following snippet is the minimum configuration you’ll need to create a KubernetesPodOperator task on Astro:

1 from airflow.configuration import conf
2 from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
3 
4 namespace = conf.get("kubernetes", "NAMESPACE")
5 
6 KubernetesPodOperator(
7     namespace=namespace,
8     image="<your-docker-image>",
9     cmds=["<commands-for-image>"],
10     arguments=["<arguments-for-image>"],
11     labels={"<pod-label>": "<label-name>"},
12     name="<pod-name>",
13     task_id="<task-name>",
14     get_logs=True,
15     in_cluster=True,
16 )

For each instantiation of the KubernetesPodOperator, you must specify the following values:

namespace = conf.get("kubernetes", "NAMESPACE"): Every Deployment runs on its own Kubernetes namespace within a cluster. Information about this namespace can be programmatically imported as long as you set this variable.
image: This is the Docker image that the operator will use to run its defined task, commands, and arguments. Astro assumes that this value is an image tag that’s publicly available on Docker Hub. To pull an image from a private registry, see Pull images from a Private Registry.
in_cluster: If a Connection object is not passed to the KubernetesPodOperator’s kubernetes_conn_id parameter, specify in_cluster=True to run the task in the Deployment’s Astro cluster.

Add the Google Artifact Registry URI

When you configure an instantiation of the KubernetesPodOperator, replace <your-docker-image> with the Google Artifact Registry image URI. To retrieve the URI:

In the Google Artifact Registry, click the registry containing the image.
Click the image you want to use.
Click the copy icon next to the image in the top corner. The string you copy should be in the format <GCP Region>-docker.pkg.dev/<Project Name>/<Registry Name>/<Image Name>.