Skip to main content
Version: 0.32

Using registry backends in Astronomer Software

Astronomer Software requires a Docker Registry to store the Docker Images generated every time a user pushes code or makes a configuration change to an Airflow Deployment on Astronomer.

The default storage backend for this Docker Registry is a Kubernetes Persistent Volume. While this may be sufficient for teams just getting started on Astronomer, Astronomer recommends backing the registry with an external storage solution for any team running in production.

The following are the registry backend tools supported by Astronomer:

Note that this doc explains only how to set up a registry for hosting Astronomer's system images. To create a custom registry for Deployment images, see Configure a custom image registry for Deployment images.

Google Cloud Storage

If you're running Astronomer Software on Google Cloud Platform (GCP) Google Kubernetes Engine (GKE), Astronomer recommends using Google Cloud Storage (GCS) as a registry backend solution.

Prerequisites

To use GCS as a registry backend solution, you'll need:

  • An existing GCS Bucket
  • Your Google Cloud Platform service account JSON Key
  • Permissions to create a Kubernetes Secret in your cluster

Update your config.yaml file

  1. Download your GCP service account JSON key from the Google Console. Make sure the service account you use has both the Storage Legacy Bucket Owner and Storage Object Admin roles.

  2. Create a Kubernetes Secret using the downloaded key:

kubectl create secret generic astronomer-gcs-keyfile --from-file astronomer-gcs-keyfile=/path/to/key.json -n <your-namespace>
  1. Add the following to your config.yaml file:
astronomer:
registry:
gcs:
enabled: true
bucket: my-gcs-bucket

Example:

#################################
## Astronomer global configuration
#################################
global:
# Base domain for all subdomains exposed through ingress
baseDomain: astro.mydomain.com

# Name of secret containing TLS certificate
tlsSecret: astronomer-tls

#################################
## Nginx configuration
#################################
nginx:
# IP address the nginx ingress should bind to
loadBalancerIP: 0.0.0.0
preserveSourceIP: true

#################################
## SMTP configuration
#################################
astronomer:
houston:
config:
email:
enabled: true
smtpUrl: YOUR_URI_HERE
registry:
gcs:
enabled: true
bucket: my-gcs-bucket
  1. Push the configuration change to your platform as described in Apply a config change.

AWS S3

If you're running Astronomer Software on the Amazon Elastic Kubernetes Service (EKS), Astronomer recommends using AWS S3 as a registry backend solution.

Prerequisites

To use AWS S3 as a registry backend solution, you'll need:

  • An S3 bucket
  • Your AWS Access Key
  • Your AWS Secret Key
  • Ability to create a Kubernetes Secret in your cluster

Create S3 IAM policy and user

  1. Use the following definition to create a new AWS IAM policy, making sure to replace S3_BUCKET_NAME with your own S3 bucket's name:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::S3_BUCKET_NAME"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload"
],
"Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
}
]
}
  1. Create a new IAM User and attach the Policy. Your access key and secret key are generated and displayed after you create the user.

  2. Create Kubernetes secrets for your key credentials in your Astronomer installation:

    $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
    $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>
  3. Select one of the following options:

  • To authenticate to AWS with your registry credentials, add this entry to the config.yaml file:
  astronomer:
registry:
s3:
enabled: true
region: us-east-1
regionendpoint: <your-region-endpoint>
bucket: <your-bucket-name>
extraEnvVars:
- name: REGISTRY_STORAGE_S3_REGION
value: <your-s3-region>
- name: REGISTRY_STORAGE_S3_ACCESSKEY
valueFrom:
secretKeyRef:
name: astronomer-s3-access-key
key: AWS_ACCESS_KEY_ID
- name: REGISTRY_STORAGE_S3_SECRETKEY
valueFrom:
secretKeyRef:
name: astronomer-s3-secret-key
key: AWS_ACCESS_SECRET_ACCESS_KEY
  • To authenticate to AWS without providing your registry credentials, add this entry to the config.yaml file:
  astronomer:
registry:
s3:
enabled: true
region: us-east-1
regionendpoint: <your-region-endpoint>
bucket: <your-bucket-name>
extraEnvVars:
- name: REGISTRY_STORAGE_S3_REGION
value: <your-s3-region>
  1. Push the configuration change to your platform. See Apply a config change.

Enable encryption (Optional)

  1. Create a key in AWS Key Management Service (KMS). During the key creation process you'll be asked to add "key users". Add the user created above as a "key user".

  2. Create Kubernetes secrets for your key credentials:

    $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
    $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>
  3. Add the following values to your config.yaml file to enable encryption:

astronomer:
registry:
s3:
enabled: true
region: us-east-1
bucket: my-s3-bucket
encrypt: true
keyid: my-kms-key-id
extraEnvVars:
- name: REGISTRY_STORAGE_S3_REGION
value: <your-s3-region>
- name: REGISTRY_STORAGE_S3_ACCESSKEY
valueFrom:
secretKeyRef:
name: astronomer-s3-access-key
key: accesskey
- name: REGISTRY_STORAGE_S3_SECRETKEY
valueFrom:
secretKeyRef:
name: astronomer-s3-secret-key
key: secretkey
  1. Push the configuration change to your platform. See Apply a config change.

Azure Blob Storage

If you're running Astronomer Software on Azure Kubernetes Service (AKS), Astronomer recommends using Azure Blob Storage as a registry backend solution.

Prerequisites

To use Azure Blog Storage as a registry backend solution, you'll need:

  • Azure Storage Account Name
  • Azure Account Access Key
  • Azure Container Name

Configure the registry backend

  1. Create Kubernetes secrets for your key credentials:

    $ kubectl create secret generic astronomer-azure-access-key --from-literal=accountname=<your-account-name> -n <your-namespace>
    $ kubectl create secret generic astronomer-azure-secret-key --from-literal=accountkey=<your-account-key> -n <your-namespace>
  2. Add the following to your config.yaml file:

astronomer:
registry:
azure:
enabled: true
accountname: my-account-name
accountkey: my-account-key
container: my-container-name
realm: core.windows.net
extraEnvVars:
- name: REGISTRY_STORAGE_AZURE_REGION
value: <your-azure-region>
- name: REGISTRY_STORAGE_AZURE_ACCOUNTNAME
valueFrom:
secretKeyRef:
name: astronomer-azure-access-key
key: accountname
- name: REGISTRY_STORAGE_AZURE_ACCOUNTKEY
valueFrom:
secretKeyRef:
name: astronomer-azure-secret-key
key: accountkey
  1. Push the configuration change to your platform as described in Apply a config change.

Was this page helpful?

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.