Using external registry backends in Astronomer Software

Astronomer Software requires a Docker Registry to store the Docker Images generated every time a user pushes code or makes a configuration change to an Airflow Deployment on Astronomer.

The default storage backend for this Docker Registry is a Kubernetes Persistent Volume. While this may be sufficient for teams just getting started on Astronomer, Astronomer recommends backing the registry with an external storage solution for any team running in production.

The following are the registry backend tools supported by Astronomer:

• Google Cloud Storage
• AWS S3
• Azure Blob Storage

Google Cloud Storage

If you’re running Astronomer Software on Google Cloud Platform (GCP) Google Kubernetes Engine (GKE), Astronomer recommends using Google Cloud Storage (GCS) as a registry backend solution.

Prerequisites

To use GCS as a registry backend solution, you’ll need:

• An existing GCS Bucket
• Your Google Cloud Platform service account JSON Key
• Permissions to create a Kubernetes Secret in your cluster

Update your values.yaml file

1. Download your GCP service account JSON key from the Google Console. Make sure the service account you use has both the Storage Legacy Bucket Owner and Storage Object Admin roles.

2. Create a Kubernetes Secret using the downloaded key:

kubectl create secret generic astronomer-gcs-keyfile --from-file astronomer-gcs-keyfile=/path/to/key.json -n <your-namespace>

3. Add the following to your values.yaml file:

1
astronomer:
2
  registry:
3
    gcs:
4
      enabled: true
5
      bucket: my-gcs-bucket

Example:

1
#################################
2
## Astronomer global configuration
3
#################################
4
global:
5
  # Base domain for all subdomains exposed through ingress
6
  baseDomain: astro.mydomain.com
7
8
  # Name of secret containing TLS certificate
9
  tlsSecret: astronomer-tls
10
11
#################################
12
## Nginx configuration
13
#################################
14
nginx:
15
  # IP address the nginx ingress should bind to
16
  loadBalancerIP: 0.0.0.0
17
  preserveSourceIP: true
18
19
#################################
20
## SMTP configuration
21
#################################
22
astronomer:
23
  houston:
24
    config:
25
      email:
26
        enabled: true
27
        smtpUrl: YOUR_URI_HERE
28
  registry:
29
    gcs:
30
      enabled: true
31
      bucket: my-gcs-bucket

4. Push the configuration change to your platform as described in Apply a config change.

AWS S3

If you’re running Astronomer Software on the Amazon Elastic Kubernetes Service (EKS), Astronomer recommends using AWS S3 as a registry backend solution.

Prerequisites

To use AWS S3 as a registry backend solution, you’ll need:

• An S3 bucket
• Your AWS Access Key
• Your AWS Secret Key
• Ability to create a Kubernetes Secret in your cluster

Create S3 IAM policy and user

1. Use the following definition to create a new AWS IAM policy, making sure to replace S3_BUCKET_NAME with your own S3 bucket’s name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
    }
  ]
}

2. Create a new IAM User and attach the Policy. Your access key and secret key are generated and displayed after you create the user.

3. Create Kubernetes secrets for your key credentials in your Astronomer installation:

   1
   $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
   2
   $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>

4. Select one of the following options:

• To authenticate to AWS with your registry credentials, add this entry to the values.yaml file:

1
  astronomer:
2
    registry:
3
      s3:
4
        enabled: true
5
        region: us-east-1
6
        regionendpoint: <your-region-endpoint>
7
        bucket: <your-bucket-name>
8
      extraEnvVars:
9
        - name: REGISTRY_STORAGE_S3_REGION
10
          value: <your-s3-region>
11
        - name: REGISTRY_STORAGE_S3_ACCESSKEY
12
            valueFrom:
13
              secretKeyRef:
14
                name: astronomer-s3-access-key
15
                key: AWS_ACCESS_KEY_ID
16
        - name: REGISTRY_STORAGE_S3_SECRETKEY
17
            valueFrom:
18
              secretKeyRef:
19
                name: astronomer-s3-secret-key
20
                key: AWS_ACCESS_SECRET_ACCESS_KEY

• To authenticate to AWS without providing your registry credentials, add this entry to the values.yaml file:

1
  astronomer:
2
    registry:
3
      s3:
4
        enabled: true
5
        region: us-east-1
6
        regionendpoint: <your-region-endpoint>
7
        bucket: <your-bucket-name>
8
      extraEnvVars:
9
        - name: REGISTRY_STORAGE_S3_REGION
10
          value: <your-s3-region>

4. Push the configuration change to your platform. See Apply a config change.

Enable encryption (Optional)

1. Create a key in AWS Key Management Service (KMS). During the key creation process you’ll be asked to add “key users”. Add the user created above as a “key user”.

2. Create Kubernetes secrets for your key credentials:

   1
   $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
   2
   $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>

3. Add the following values to your values.yaml file to enable encryption:

1
astronomer:
2
  registry:
3
    s3:
4
      enabled: true
5
      region: us-east-1
6
      bucket: my-s3-bucket
7
      encrypt: true
8
      keyid: my-kms-key-id
9
    extraEnvVars:
10
      - name: REGISTRY_STORAGE_S3_REGION
11
        value: <your-s3-region>
12
      - name: REGISTRY_STORAGE_S3_ACCESSKEY
13
          valueFrom:
14
            secretKeyRef:
15
              name: astronomer-s3-access-key
16
              key: accesskey
17
      - name: REGISTRY_STORAGE_S3_SECRETKEY
18
          valueFrom:
19
            secretKeyRef:
20
              name: astronomer-s3-secret-key
21
              key: secretkey

3. Push the configuration change to your platform. See Apply a config change.

Authorize Astronomer to a registry backend using IAM roles (Optional)

To avoid hardcoding credentials for your registry backend, add the following configuration to your values.yaml file:

1
registry:
2
    serviceAccount:
3
      # Specifies whether a service account should be created
4
      create: true
5
      # Annotations to add to the service account
6
      annotations:
7
        eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxxxx:role/<your-iam-role>
8
    s3:
9
      enabled: true
10
      region: <your-region>
11
      bucket: <your-registry-backend>

Then, push the configuration change to your platform. See Apply a config change.

Azure Blob Storage

If you’re running Astronomer Software on Azure Kubernetes Service (AKS), Astronomer recommends using Azure Blob Storage as a registry backend solution.

Prerequisites

To use Azure Blog Storage as a registry backend solution, you’ll need:

• Azure Storage Account Name
• Azure Account Access Key
• Azure Container Name

Configure the registry backend

1. Create Kubernetes secrets for your key credentials:

   1
   $ kubectl create secret generic astronomer-azure-access-key --from-literal=accountname=<your-account-name> -n <your-namespace>
   2
   $ kubectl create secret generic astronomer-azure-secret-key --from-literal=accountkey=<your-account-key> -n <your-namespace>

2. Add the following to your values.yaml file:

1
astronomer:
2
  registry:
3
    azure:
4
      enabled: true
5
      accountname: my-account-name
6
      accountkey: my-account-key
7
      container: my-container-name
8
      realm: core.windows.net
9
    extraEnvVars:
10
      - name: REGISTRY_STORAGE_AZURE_REGION
11
        value: <your-azure-region>
12
      - name: REGISTRY_STORAGE_AZURE_ACCOUNTNAME
13
          valueFrom:
14
            secretKeyRef:
15
              name: astronomer-azure-access-key
16
              key: accountname
17
      - name: REGISTRY_STORAGE_AZURE_ACCOUNTKEY
18
          valueFrom:
19
            secretKeyRef:
20
              name: astronomer-azure-secret-key
21
              key: accountkey

3. Push the configuration change to your platform as described in Apply a config change.