AdministrationInstall

Install Astronomer Software

This guide describes the steps to install Astronomer Software, which allows you to deploy and scale any number of Apache Airflow deployments.

Astronomer recommends platform administrators perform their first Astronomer Software installation manually using the procedures described in this document. You can adapt the procedure to meet organizational best practices when deploying to change-controlled environments.

Prerequisites

The following prerequisites apply when running Astronomer Software on Amazon EKS. See the Other tab if you run a different version of Kubernetes on AWS.
  • An EKS Kubernetes cluster, running a version of Kubernetes certified as compatible on the Version Compatibility Reference that provides the following components:
  • A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the Version Compatibility Reference.
  • PostgreSQL superuser permissions.
  • Permission to create and modify resources on AWS.
  • Permission to generate a certificate that covers a defined set of subdomains.
  • An SMTP service and credentials. For example, Mailgun or Sendgrid.
  • The AWS CLI.
  • (Optional) eksctl for creating and managing your Astronomer cluster on EKS.
  • A machine meeting the following criteria with access to the Kubernetes API Server:
    • Network access to the Kubernetes API Server - either direct access or VPN.
    • Network access to load-balancer resources that are created when Astronomer Software is installed later in the procedure - either direct access or VPN.
    • Configured to use the DNS servers where Astronomer Software DNS records can be created.
    • Helm (minimum v3.6).
    • The Kubernetes CLI (kubectl).
  • (Situational) The OpenSSL CLI might be required to troubleshoot certain certificate-related conditions.

Step 1: Plan the structure of your Astronomer Software environments

Before installing Astronomer Software, consider how many instances of the platform you want to host because you install each of these instances on separate Kubernetes clusters following the instructions in this document.

Each instance of Astronomer Software can host multiple Airflow environments, or Deployments. Some common types of Astronomer Software instances you might consider hosting are:

  • Sandbox: The lowest environment that contains no sensitive data, used only by system-administrators to experiment, and not subject to change control.
  • Development: User-accessible environment that is subject to most of the same restrictions of higher environments, with relaxed change control rules.
  • Staging: All network, security, and patch versions are maintained at the same level as in the production environment. However, it provides no availability guarantees and includes relaxed change control rules.
  • Production: The production instance hosts your production Airflow environments. You can choose to host development Airflow environments here or in environments with lower levels of support and restrictions.

For each instance of the platform that you plan to host, create a project folder to contain the platform instance’s configurations. For example, if you want to install a development environment, create a folder named ~/astronomer-dev.

Certain files in the project directory might contain secrets when you set up your sandbox or development environments. For your first install, keep these secrets in a secure place on a suitable machine. As you progress to higher environments, such as staging or production, secure these files separately in a vault and use the remaining project files in your directory to serve as the basis for your CI/CD deployment.

Step 2: Create values.yaml from a template

Astronomer Software uses Helm to apply platform-level configurations. Use one of the following templates as the basis for your Astronomer Software platform configuration by copying the template into a local file named values.yaml in your platform project directory. As you continue to extend the functionality of your Astronomer Software instances, you can continually modify this file to take advantage of new features.

As you copy the template configuration, keep the following in mind.

  • Do not make any changes to this file until instructed to do so in later steps.
  • Do not run helm upgrade or upgrade.sh until instructed to do so in later steps.
  • Ignore any instructions to run helm upgrade from other Astronomer documentation until you’ve completed this installation.
1#################################
2### Astronomer global configuration for EKS
3#################################
4global:
5  # Base domain for all subdomains exposed through ingress
6  baseDomain: sandbox-astro.example.com
7
8  # Name of secret containing TLS certificate
9  tlsSecret: astronomer-tls
10
11  # List of secrets containing the cert.pem of trusted private certification authorities
12  # Example command: `kubectl create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
13  # privateCaCerts:
14  # - private-root-ca
15
16  # For development or proof-of-concept, you can use an in-cluster database
17  # postgresqlEnabled: true is NOT supported in production.
18  postgresqlEnabled: false
19
20  ssl:
21    # if doing a proof-of-concept with in-cluster-db, this must be set to false
22    enabled: true
23  dagOnlyDeployment:
24    enabled: true
25  enableHoustonInternalAuthorization: true # Access houston authorization without traverssing the external load-balanccer
26
27#################################
28### Nginx configuration
29#################################
30nginx:
31  # IP address the nginx ingress should bind to
32  loadBalancerIP: ~
33  #  set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
34  privateLoadBalancer: true
35  # Dictionary of arbitrary annotations to add to the nginx ingress. For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
36  ingressAnnotations: {service.beta.kubernetes.io/aws-load-balancer-type: nlb} # Change to 'elb' if your node group is private and doesn't utilize a NAT gateway
37  # If all subnets are private, auto-discovery may fail.
38  # You must enter the subnet IDs manually in the annotation below.
39  # service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-id-1,subnet-id-2
40astronomer:
41  houston:
42    config:
43      publicSignups: false # Users need to be invited to have access to Astronomer. Set to true otherwise
44      emailConfirmation: true # Users get an email verification before accessing Astronomer
45      deployments:
46        hardDeleteDeployment: true # Allow deletions to immediately remove the database and namespace
47        manualReleaseNames: true # Allows you to set your release names
48        serviceAccountAnnotationKey: eks.amazonaws.com/role-arn # Flag to enable using IAM roles (don't enter a specific role)
49        configureDagDeployment: true # Required for dag-only deploys
50        enableUpdateDeploymentImageEndpoint: true # Enables apis for deploying images
51        upsertDeploymentEnabled: true # Enables additional apis for updating deployments
52      email:
53        enabled: true
54        reply: "noreply@my.email.internal" # Emails will be sent from this address
55      auth:
56        github:
57          enabled: true # Lets users authenticate with Github
58        local:
59          enabled: false # Disables logging in with just a username and password
60        openidConnect:
61          google:
62            enabled: true # Lets users authenticate with Google
63    upgradeDeployments:
64      enabled: false # dont automatically upgrade airflow instances when the platform is upgraded
65    secret:
66    - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Can be removed if email is not used.
67      secretName: "astronomer-smtp"
68      secretKey: "connection"
69  commander:
70    airGapped:
71      enabled: true # leave true even if not airgapped

Step 3: Decide whether to use a third-party Ingress controller

Astronomer Software requires a Kubernetes Ingress controller to function and provides an integrated Ingress controller by default.

Astronomer generally recommends you use the integrated Ingress controller, but Astronomer Software also supports certain third-party ingress-controllers.

Ingress controllers typically need elevated permissions (including a ClusterRole) to function. Specifically, the Astronomer Software Ingress controller requires the ability to:

  • List all namespaces in the cluster.
  • View ingresses in the namespaces.
  • Retrieve secrets in the namespaces to locate and use private TLS certificates that service the ingresses.

If you have complex regulatory requirements, you might need to use an Ingress controller that’s approved by your organization and disable Astronomer’s integrated controller. You configure this detail later in the installation.

Step 4: Choose and configure a base domain

When you install Astronomer Software, it creates a variety of services that your users access to manage, monitor, and run Airflow.

Choose a base domain such as astronomer.example.com, astro-sandbox.example.com, astro-prod.example.internal for which:

  • You have the ability to create and edit DNS records
  • You have the ability to issue TLS certificates
  • The following addresses are available:
    • app.<base-domain>
    • deployments.<base-domain>
    • houston.<base-domain>
    • grafana.<base-domain>
    • kibana.<base-domain>
    • install.<base-domain>
    • alertmanager.<base-domain>
    • prometheus.<base-domain>
    • registry.<base-domain>

The base domain itself does not need to be available and can point to another service not associated with Astronomer or Airflow. If the base domain is available, you can choose to establish a vanity redirect from <base-domain> to app.<base-domain> later in the installation process.

When choosing a base domain, consider the following:

  • The name you choose must be be resolvable by both your users and Kubernetes itself.
  • You need to have or obtain a TLS certificate that is recognized as valid by your users. If you use the Astronomer Software integrated container registry, the TLS certification must also be recognized as valid by Kubernetes itself.
  • Wildcard certificates are only valid one level deep. For example, an ingress controller that uses a certificate called *.example.com can provide service for app.example.com but not app.astronomer-dev.example.com.
  • The bottom-level hostnames, such as app, registry, prometheus, are fixed and cannot be changed.
  • Most Kubernetes clusters don’t resolve DNS hostnames with more than five segments, each separated by the dot character, .. For example, app.astronomer.sandbox.mygroup.example.com is six segments and might cause problems. Astronomer recommends choosing a base domain of astronomer-sandbox.mygroup.example.com instead of astronomer.sandbox.mygroup.example.com.

The base domain is visible to end users. They can view the base domain in the following scenarios:

  • When users access the Astronomer Software UI. For example, https://app.sandbox-astro.example.com.
  • When users access an Airflow Deployment. For example, https://deployments.sandbox-astro.example.com/deployment-release-name/airflow.
  • When users authenticate to the Astro CLI. For example, astro login sandbox-astro.example.com.
If you install Astronomer Software on OpenShift and also want to use OpenShift’s integrated ingress controller, you can use the the hostname of the default OpenShift ingress controller as your base domain, such as app.apps.<OpenShift-domain>. Doing this requires permission to reconfigure the route admission policy for the standard ingress controller to InterNamespaceAllowed. See Third Party Ingress Controller - Configuration notes for OpenShift for additional information and options.

Configure the base domain

Locate the global.baseDomain in your values.yaml file and change it to your base domain as shown in the following example:

1global:
2  # Base domain for all subdomains exposed through ingress
3  baseDomain: sandbox-astro.example.com

Step 5: Create the Astronomer Software platform namespace

In your Kubernetes cluster, create a Kubernetes namespace to contain the Astronomer Software platform. Astronomer recommends naming this namespace astronomer.

1kubectl create namespace astronomer

Astronomer Software uses the contents of this namespace to provision and manage Airflow instances running in other namespaces. Each Airflow instance has its own isolated namespace.

Step 6: (Optional) Configure third-party Ingress controller DNS

Skip this step if you use Astronomer Software’s integrated ingress controller.

Follow the instructions in this section to create DNS entries pointing to the third-party ingress controller instance that provides ingress service to Astronomer Software.

Astronomer Software Third-Party DNS Requirements and Record Guidance

Astronomer Software requires the following domain names be registered and resolvable within the Kubernetes cluster and to Astronomer Software users:

  • app.<base-domain> (required)
  • deployments.<base-domain> (required)
  • houston.<base-domain> (required)
  • prometheus.<base-domain> (required)
  • grafana.<base-domain> (required if using Astronomer Software’s integrated grafana)
  • kibana.<base-domain> (required if not using external elasticsearch)
  • registry.<base-domain> (required if using Astronomer Software’s integrated container-registry)
  • alertmanager.<base-domain> (required if using Astronomer Software’s integrated Alert Manager)
  • <base-domain> (optional but recommended, provides a vanity re-direct to app.<base-domain>)
  • install.<base-domain> (optional)

Astronomer generally recommends that:

  • The <base-domain> record is a zone apex record, that is typically expressed by using a hostname of @, and points to the IP(s) of the ingress controller.
  • All other records are CNAME records that point to the <base-domain>.

For platform administrators unable to register the base domain, Astronomer recommends that:

  • The app.<baseDomain> record is a record pointing to the IPs of the ingress controller.
  • All other records are CNAME records pointing to app.<base-domain>.
For lower environments like sandboxes or development environments, Astronomer recommends a relatively short ttl value, such as 60 seconds, when you first deploy Astronomer so that any errors can be quickly corrected.

Request ingress information from your ingress administrator

Provide the details you gathered in Astronomer Software Third-Party DNS Requirements and Record Guidance to your ingress controller administrator, making sure to replace <base-domain> with your chosen base domain. Then, request the following information from the administrator:

  • The ingress class name to use, or whether you should leave the class name blank and use the default.
  • The IP addresses to use for DNS entries pointing to the ingress controller.
  • Whether DNS records are automatically created in response to ingress sources that you create later in the install.
  • Whether DNS records need to be manually created, and if so, who coordinates their creation and who creates them.

Save this information for later in this setup.

Create DNS records pointing to your third-party Ingress controller

Create DNS records that point to your third-party ingress controller based on your organization’s standard workflows.

Use dig <hostname> or getent hosts <hostname> to verify that each DNS entry is created and points to the IP address of the ingress controller you use.

Step 7: Request and validate an Astronomer TLS certificate

To install Astronomer Software, you need a TLS certificate that is valid for several domains. One of the domains is the primary name on the certificate, also known as the common name (CN). The additional domains are equally valid, supplementary domains known as Subject Alternative Names (SANs).

Astronomer requires a private certificate to be present in the Astronomer Software platform namespace, even if you use a third-party ingress controller that doesn’t otherwise require it.

Request an ingress controller TLS certificate

Request a TLS certificate from your security team for Astronomer Software. In your request, include the following:

  • Your chosen base domain as the Common Name (CN). If your certificate authority will not issue certificates for the bare base domain, use app.<base-domain> as the CN instead.
  • Either a wildcard Subject Alternative Name (SAN) entry of *.<base-domain> or an explicit SAN entry for each of the following items:
    • app.<base-domain> (omit if already used as the Common Name)
    • deployments.<base-domain>
    • registry.<base-domain>
    • houston.<base-domain>
    • grafana.<base-domain>
    • kibana.<base-domain>
    • install.<base-domain>
    • alertmanager.<base-domain>
    • prometheus.<base-domain>
  • If you use the Astronomer Software integrated container registry, specify that that the encryption type of the certificate must be RSA.
  • Request the following return format:
    • A key.pem containing the private key in pem format
    • Either a full-chain.pem (containing the public certificate additional certificates required to validate it, in pem format) or a bare cert.pem and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
    • Either the private-root-ca.pem in pem format of the private Certificate Authority used to create your certificate or a statement that the certificate is signed by a public Certificate Authority.
If you’re using the Astronomer Software integrated container registry, the encryption type used on your TLS certificate must be RSA. Cerbot users must include -key-type rsa when requesting certificates. Most other solutions generate RSA keys by default.

Validate the received certificate and associated items

Ensure that you received each of the following three items:

  • A key.pem containing the private key in pem format.
  • Either a full-chain.pem, in pem format, that contains the public certificate additional certificates required to validate it or a bare cert.pem and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
  • Either the private-root-ca.pem in pem format of the the private Certificate Authority used to create your certificate or a statement that the certificate is signed by public Certificate Authority.

To validate that your security team generated the correct certificate, run the following command using the openssl CLI:

1openssl x509 -in  <your-certificate-filepath> -text -noout

This command will generate a report. If the X509v3 Subject Alternative Name section of this report includes either a single *.<base-domain> wildcard domain or all subdomains, then the certificate creation was successful.

Confirm that your full-chain certificate chain is ordered correctly. To determine your certificate chain order, run the following command using the openssl CLI:

1openssl crl2pkcs7 -nocrl -certfile <your-full-chain-certificate-filepath> | openssl pkcs7 -print_certs -noout

The command generates a report of all certificates. Verify that the certificates are in the following order:

  • Domain
  • Intermediate (optional)
  • Root

(Optional) Additional validation for the Astronomer integrated container registry

If you don’t plan to store images in Astronomer’s integrated container registry and instead plan to store all container images using an external container registry, you can skip this step.

The Astronomer Software integrated container registry requires that your private key signs traffic originating from the Astronomer Software platform using the RSA encryption method. Confirm that the key is signing traffic correctly before proceeding.

Run the following command to extract the bare public cert, if it was not already included in the files provided by your certificate authority, from the full-chain certificate file:

1openssl crl2pkcs7 -nocrl -certfile full-chain.pem | openssl pkcs7 -print_certs -noout > cert.pem

Examine the public certificate and ensure all Signature Algorithms are listed as sha1WithRSAEncryption.

1openssl x509 -in cert.pem -text|grep Algorithm
2        Signature Algorithm: sha1WithRSAEncryption
3            Public Key Algorithm: rsaEncryption
4    Signature Algorithm: sha1WithRSAEncryption

If your key is not compatible with the Astronomer Software integrated container registry, ask your Certificate Authority to re-issue the credentials and emphasize the need for an RSA cert, or use an external container registry.

Step 8: Store and configure the ingress controller public TLS full-chain certificate

Run the following command to store the public full-chain certificate in the Astronomer Software Platform Namespace in a tls-type Kubernetes secret. You can create a custom name for this secret. The following example uses the name, astronomer-tls.

1kubectl -n <astronomer platform namespace> create secret tls astronomer-tls --cert <fullchain-pem-filepath> --key <your-private-key-filepath>

However, if your security team has instructed you that there are no intermediate certificates, run the following command.

1kubectl -n astronomer create secret tls astronomer-tls --cert full-chain.pem --key server_private_key.pem

Naming the secret astronomer-tls with no substitutions is recommended when using a third-party ingress controller.

Step 9: (Optional) Configure a third-party ingress controller

If you use Astronomer Software’s integrated ingress controller, you can skip this step.

Complete the full setup as described in Third-party Ingress-Controllers, which includes steps to configure ingress controllers in specific environment types. When you’re done, return to this page and continue to the next step.

Step 10: Configure a private certificate authority

Skip this step if you don’t use a private Certificate Authority (private CA) to sign the certificate used by your ingress-controller. Or, if you don’t use a private CA for any of the following services that the Astronomer Software platform interacts with.

Astronomer Software trusts public Certificate Authorities automatically.

Astronomer Software must be configured to trust any private Certificate Authorities issuing certificates for systems Astronomer Software interacts with, including but not limited-to:

  • ingress controller
  • email server, unless disabled
  • any container registries that Kubernetes pulls from
  • if using OAUTH, the OAUTH provider
  • if using external elasticsearch, any external elasticsearch instances
  • if using external prometheus, any external prometheus instances

Perform the procedure described in Configuring private CAs for each certificate authority used to sign TLS certificates.

Astro CLI users must also configure both their operating system and container solution, Docker Desktop or Podman, to trust the private certificate Authority that was used to create the certificate used by the Astronomer Software ingress controller and any third-party container registries.

Step 11: Confirm your Kubernetes cluster trusts required CAs

If at least one of the following circumstances apply to your installation, complete this step:

  • You configured Astronomer Software to pull platform container images from an external container registry that uses a certificate signed by a private CA.
  • You plan for your users to deploy Airflow images to Astronomer Software’s integrated container registry and Astronomer is using a TLS certificate issued by a private CA.
  • Users will deploy images to an external container registry and that registry is using a TLS certificate issued by a private CA.

Kubernetes must be able to pull images from one or more container registries for Astronomer Software to function. By default, Kubernetes only trusts publicly signed certificates. This means that by default, Kubernetes does not honor the list of certificates trusted by the Astronomer Software platform.

Many enterprises configure Kubernetes to trust additional certificate authorities as part of their standard cluster creation procedure. Contact your Kubernetes Administrator to find out what, if any, private certificates are currently trusted by your Kubernetes Cluster. Then, consult your Kubernetes administrator and Kubernetes provider’s documentation for instructions on configuring Kubernetes to trust additional CAs.

Follow procedures for your Kubernetes provider to configure Kubernetes to trust each CA associated with your container registries, including the integrated container registry, if applicable.

Certain clusters do not provide a mechanism to configure the list of certificates trusted by Kubernetes.

While configuring the Kubernetes list of cluster certificates is a customer responsibility, Astronomer Software includes an optional component that can, for certain Kubernetes cluster configurations, add certificates defined in global.privateCaCerts to the list of certificates trusted by Kubernetes. This can be enabled by setting global.privateCaCertsAddToHost.enabled and global.privateCaCertsAddToHost.addToContainerd to true in your values.yaml file and setting global.privateCaCertsAddToHost.containerdConfigToml to:

[host."https://registry.<baseApp>"]
  ca = "/etc/containerd/certs.d/<registry hostname>/<secret name>.pem"

For example, if your base domain is astro-sandbox.example.com and the CA public-certifice certificate is stored in the platform namespace in a secret named my-private-ca, the global.privateCaCertsAddToHost section would be:

1  global:
2    privateCaCertsAddToHost:
3      enabled: true
4      addToContainerd: true
5      hostDirectory: /etc/containerd/certs.d
6      containerdConfigToml: |-
7        [host."https://registry.astro-sandbox.example.com"]
8          ca = "/etc/containerd/certs.d/registry.astro-sandbox.example.com/my-private-ca.pem"

Step 12: Configure outbound SMTP email

Astronomer Software requires the ability to send email to:

  • Notify users of errors with their Airflow Deployments.
  • Send emails to invite new users to Astronomer.
  • Send certain platform alerts, enabled by default but can be configured.

Astronomer Software sends all outbound email using SMTP.

If SMTP is not available in the environment where you’re installing Astronomer Software, follow instructions in configure Astronomer Software to not send outbound email, and then skip the rest of this section.

  1. Obtain a set of SMTP credentials from your email administrator for you to use to send email from Astronomer Software. When you request an email address and display name, remember that these emails are not designed for users to reply directly to them. Request all the following information:

    • Email address
    • Email display name requirements. Some email servers require a From line of: Do Not Reply <donotreply@example.com>.
    • SMTP username. This is usually the same as the email address.
    • SMTP password
    • SMTP hostname
    • SMTP port
    • Whether or not the connection supports TLS

    If there is a / or any other escape character in your username or password, you may need to URL encode those characters.

  2. Ensure that your Kubernetes cluster has access to send outbound email to the SMTP server.

  3. Change the configuration in values.yaml from noreply@my.email.internal to an email address that is valid to use with your SMTP credentials.

  4. Construct an email connection string and store it in a secret in the Astronomer platform namespace. The following example shows how to store the connection in a secret called astronomer-smtp. Make sure to url-encode the username and password if they contain special characters.

    1kubectl -n astronomer create secret generic astronomer-smtp --from-literal connection="smtp://my@40user:my%40pass@smtp.email.internal/?requireTLS=true"

    In general, an SMTP URI is formatted as smtps://USERNAME:PASSWORD@HOST/?pool=true. The following table contains examples of the URI for some of the most popular SMTP services:

    ProviderExample SMTP URL
    AWS SESsmtp://AWS_SMTP_Username:AWS_SMTP_Password@email-smtp.us-east-1.amazonaws.com/?requireTLS=true
    SendGridsmtps://apikey:SG.sometoken@smtp.sendgrid.net:465/?pool=true
    Mailgunsmtps://xyz%40example.com:password@smtp.mailgun.org/?pool=true
    Office365smtp://xyz%40example.com:password@smtp.office365.com:587/?requireTLS=true
    Custom SMTP-relaysmtp://smtp-relay.example.com:25/?ignoreTLS=true

    If your SMTP provider is not listed, refer to the provider’s documentation for information on creating an SMTP URI.

If there is a / or any other escape character in your username or password, you may need to URL encode those characters.

Step 13: Configure volume storage classes

Skip this step if your cluster defines a volume storage class, and you want to use it for all volumes associated with Astronomer Software and its Airflow Deployments.

Astronomer strongly recommends that you do not back any volumes used for Astronomer Software with mechanical hard drives.

Replace <desired-storage-class> in the following configuration with the storage class you want to use for each respective component. You can remove the configuration for any component where using the default storage is acceptable.

1alertmanager:
2  persistence:
3    storageClassName: "<desired-storage-class>"
4stan:
5  store:
6    volume:
7      storageClass: "<desired-storage-class>"
8prometheus:
9  persistence:
10    storageClassName: "<desired-storage-class>"
11elasticsearch:
12  common:
13    persistence:
14      storageClassName: "<desired-storage-class>"
15astronomer:
16  registry:
17    persistence:
18      storageClassName: "<desired-storage-class>"
19  houston:
20    config:
21      deployments:
22        helm:
23          dagDeploy:
24            persistence:
25              storageClass: "<desired-storage-class>"
26          airflow:
27            redis:
28              persistence:
29                storageClassName: "<desired-storage-class>"
30# this option does not apply when using an external postgres database
31# bundled postgresql not a supported option, only for use in proof-of-concepts
32postgresql:
33  persistence:
34    storageClass: "<desired-storage-class>"

Merge these values into values.yaml. You can do this manually or by placing merge_yaml.py and the configuration as a new file in your project directory and running python merge_yaml.py storage-class-config.yaml values.yaml.

Step 14: Configure the database

Astronomer requires a central Postgres database that acts as the backend for Astronomer’s Houston API and hosts individual metadata databases for all Deployments created on the platform.

If, while evaluating Astronomer Software, you need to create a temporary environment where Postgres is not available, locate the global.postgresqlEnabled option already present in your values.yaml and set it to true, then skip the remainder of this step.

Note that global.postgresqlEnabled to true is an unsupported configuration, and should never be used on any development, staging, or production environment.

If you use Azure Database for either PostgreSQL or another Postgres instance that does not enable the pg_trgm by default, you must enable the pg_trgm extension prior to installing Astronomer Software. If pg_trgm is not enabled, the install will fail. pg_tgrm is enabled by default on Amazon RDS and Google Cooud SQL for PostgresQL.

For instructions on enabling the pg_trgm extension for Azure Flexible Server, see PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server.

Additional requirements apply to the following databases:

  • AWS RDS:
    • t2 medium is the minimum RDS instance size you can use.
  • Azure Flexible Server:
    • You must enable the pg_trgm extension as per the advisory earlier in this section.
    • Set global.ssl.modeto prefer in your values.yaml file.

Create a Kubernetes Secret named astronomer-bootstrap that points to your database. You must URL encode any special characters in your Postgres password.

To create this secret, run the following command replacing the astronomer platform namespace, username, password, database hostname, and database port with their respective values. Remember that username and password must be url-encoded if they contain special-characters:

1kubectl --namespace <astronomer platform namespace> create secret generic astronomer-bootstrap \
2  --from-literal connection="postgres://<url-encoded username>:<url-encoded password>@<database hostname>:<database port>"

For example, for a username named bob with password abc@abc at hostname some.host.internal, you would run:

1kubectl --namespace astronomer create secret generic astronomer-bootstrap \
2  --from-literal connection="postgres://bob:abc%40abc@some.host.internal:5432"

Step 15: Configure an external Docker registry for Airflow images

By default, Astronomer Software users create customized Airflow container images when they deploy project code to the platform. These images frequently contain sensitive information and must be stored in a secure location accessible to Kubernetes.

Astronomer Software includes an integrated image registry for this purpose.

Users can use images hosted in other container image repositories accessible to the Kubernetes cluster without additional platform-level configuration.

See Configure a custom registry for Deployment images for additional configurable options.

Step 16: Configure the Docker registry used for platform images

Skip this step if you are installing Astronomer Software onto a Kubernetes cluster that can pull container images from public image repositories and you don’t want to mirror these images locally.

Configure your install to use a custom image repository for platform images

Astronomer expects the images to use their normal names, but prefixed by a string you define. For example, if you specify artifactory.example.com/astronomer, when you mirror images later in this procedure, you mirror quay.io/astronomer/<image> as <custom-platform-repo-prefix>/astronomer/<image>. The following show additional examples:

  • quay.io/astronomer/ap-houston-api to artifactory.example.com/astronomer/ap-houston-api
  • quay.io/astronomer/astronomer/ap-commander to artifactory.example.com/astronomer/ap-commander

Replace <custom-platform-repo-prefix> in the following configuration data with your platform image repository prefix and merge into values.yaml either manually or by placing merge_yaml.py in your astro-platform project-directory and running python merge_yaml.py private-platform-registry-snippet.yaml values.yaml.

1global:
2  privateRegistry:
3    enabled: true
4    repository: <custom-platform-repo-prefix>
5  airflow:
6    images:
7      airflow:
8        repository: <custom-platform-repo-prefix>/ap-airflow
9      statsd:
10        repository: <custom-platform-repo-prefix>/ap-statsd-exporter
11      redis:
12        repository: <custom-platform-repo-prefix>/ap-redis
13      pgbouncer:
14        repository: <custom-platform-repo-prefix>/ap-pgbouncer
15      pgbouncerExporter:
16        repository: <custom-platform-repo-prefix>/ap-pgbouncer-exporter
17      gitSync:
18        repository: <custom-platform-repo-prefix>/ap-git-sync
19astronomer:
20  houston:
21    config:
22      deployments:
23        helm:
24          runtimeImages:
25            airflow:
26              repository: <custom-platform-repo-prefix>/astro-runtime
27            flower:
28              repository: <custom-platform-repo-prefix>/astro-runtime
29          airflow:
30            defaultAirflowRepository: <custom-platform-repo-prefix>/ap-airflow
31            defaultRuntimeRepository: <custom-platform-repo-prefix>/astro-runtime

For example, if your custom platform image registry prefix was 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer, your configuration would look like the following:

1global:
2  privateRegistry:
3    enabled: true
4    repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer
5  airflow:
6    images:
7      airflow:
8        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-airflow
9      statsd:
10        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-statsd-exporter
11      redis:
12        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-redis
13      pgbouncer:
14        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-pgbouncer
15      pgbouncerExporter:
16        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-pgbouncer-exporter
17      gitSync:
18        repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-git-sync
19astronomer:
20  houston:
21    config:
22      deployments:
23        helm:
24          runtimeImages:
25            airflow:
26              repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/astro-runtime
27            flower:
28              repository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/astro-runtime
29          airflow:
30            defaultAirflowRepository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/ap-airflow
31            defaultRuntimeRepository: 012345678910.dkr.ecr.us-east-1.amazonaws.com/myrepo/astronomer/astro-runtime

Configure authentication to a custom platform registry

These instructions do not apply to images hosted on Amazon Elastic Container Registry (ECR). Credentials for ECR have a limited lifespan and are unsuitable for using on Astronomer Software. To use AWS ECR to serve images for Astronomer, you must grant permissions for the following actions to the Kubernetes Nodes IAM Role.

1  "ecr:GetDownloadUrlForLayer",
2  "ecr:BatchGetImage"

Astronomer Software platform images are usually hosted in internal repositories that do not require configuration. If your repository requires you pass an image credential:

  1. Log in to the registry and follow the Kubernetes documentation to produce a /.docker/config.json file.

  2. Run the following command to create an image pull secret named platform-regcred in the Astronomer Software platform namespace:

    1kubectl -n <astronomer platform namespace> create secret generic platform-regcred \
    2--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
    3--type=kubernetes.io/dockerconfigjson

  3. Set global.privateRegistry.secretName in values.yaml to platform-regcred. For example:

    1global:
    2  privateRegistry:
    3    secretName: platform-regcred

Step 17: Determine which version of Astronomer Software to install

Astronomer recommends new Astronomer Software installations use the most recent version available in either the Stable or Long Term Support (LTS) release-channel. Keep this version number available for the following steps.

See Astronomer Software’s lifecycle policy and version compatibility reference for more information.

Step 18: Fetch Airflow Helm charts

If you have internet access to https://helm.astronomer.io, run the following command on the machine where you want to install Astronomer Software:

1helm repo add astronomer https://helm.astronomer.io/
2helm repo update

If you don’t have internet access to https://helm.astronomer.io, download the Astronomer Software Platform Helm chart file corresponding to the version of Astronomer Software you are installing or upgrading to from https://helm.astronomer.io/astronomer-<version number>.tgz. For example, for Astronomer Software v0.36.0 you would download https://helm.astronomer.io/astronomer-0.36.0.tgz. This file does not need to be uploaded to an internal chart repository.

Step 19: Create and customize upgrade.sh

Create a file named upgrade.sh in your platform deployment project directory containing the following script. Specify the following values at the beginning of the script:

  • CHART_VERSION: Your Astronomer Software version, including patch and a v prefix. For example, v0.36.0.
  • RELEASE_NAME: Your Helm release name. astronomer is strongly recommended.
  • NAMESPACE: The namespace to install platform components into. astronomer is strongly recommended.
  • CHART_NAME: Set to astronomer/astronomer if fetching platform images from the internet. Otherwise, specify the filename if you’re installing from a file (for example astronomer-0.36.0.tgz).
1#!/bin/bash
2set -xe
3
4# typically astronomer
5RELEASE_NAME=<astronomer-platform-release-name>
6# typically astronomer
7NAMESPACE=<astronomer-platform-namespace>
8# typically astronomer/astronomer
9CHART_NAME=<chart name>
10# format is v<major>.<minor>.<path> e.g. v0.32.9
11CHART_VERSION=<v-prefixed version of the Astronomer Software platform chart>
12# ensure all the above environment variables have been set
13
14helm repo add --force-update astronomer https://helm.astronomer.io
15helm repo update
16
17# upgradeDeployments false ensures that Airflow charts are not upgraded when this script is run
18# If you deployed a config change that is intended to reconfigure something inside Airflow,
19# then you may set this value to "true" instead. When it is "true", then each Airflow chart will
20# restart. Note that some stable version upgrades require setting this value to true regardless of your own configuration.
21# If you are currently on Astronomer Software 0.25, 0.26, or 0.27, you must upgrade to version 0.28 before upgrading to 0.29. A direct upgrade to 0.29 from a version lower than 0.28 is not possible.
22helm upgrade --install --namespace $NAMESPACE \
23            -f ./values.yaml \
24            --reset-values \
25            --version $CHART_VERSION \
26            --debug \
27            --set astronomer.houston.upgradeDeployments.enabled=false \
28            $RELEASE_NAME \
29            $CHART_NAME $@

Step 20: Fetch images from Astronomer’s Helm template

  1. To gather the list of images, use one of the following methods:

Mac and Linux users with jq installed can set CHART_VERSION in the following snippet and run it to produce a list of images.

1CHART_VERSION=<v-prefixed version of the Astronomer Software platform chart>
2UNPREFIXED_CHART_VERSION=${CHART_VERSION#v}
3curl -s https://updates.astronomer.io/astronomer-software/releases/astronomer-${UNPREFIXED_CHART_VERSION}.json | jq -r '(.astronomer.images, .airflow.images) | to_entries[] | "\(.value.repository):\(.value.tag)"'| sort
  1. Copy the above images to the container registry using the naming scheme you configured when you set up a custom image registry.

Step 21: Fetch Airflow/Astro Runtime updates

If you are not installing Astronomer Software into an egress-controlled or airgapped environment, skip this step.

By default, Astronomer Software checks for Airflow updates, which are included in the Astro Runtime, once per day at midnight by querying https://updates.astronomer.io/astronomer-runtime. This returns a JSON file with details about the latest available Astro Runtime versions.

To store this in the cluster itself, and avoid this check, complete the following steps:

  1. Download the JSON files and store them in a Kubernetes configmap by running the following commands:

    1$ curl -L https://updates.astronomer.io/astronomer-certified --output astronomer-certified.json
    2$ curl -L https://updates.astronomer.io/astronomer-runtime --output astronomer-runtime.json
    3$ kubectl create configmap astronomer-certified --from-file=astronomer-certified.json=./astronomer-certified.json -n astronomer
    4$ kubectl create configmap astronomer-runtime --from-file=astronomer-runtime.json=./astronomer-runtime.json -n astronomer

  2. Add the following values into values.yaml. You can do this manually or by placing the configuration as a new file, along with merge_yaml.py in your project directory and running python merge_yaml.py openshift.yaml values.yaml.

    1astronomer:
    2  houston:
    3    updateCheck:
    4      enabled: false
    5    updateAirflowCheck:
    6      url: http://127.0.0.1/disabled-by-config
    7    updateRuntimeCheck:
    8      url: http://127.0.0.1/disabled-by-config
    9    config:
    10      deployments:
    11        airflowReleasesFile: '/certified-releases/local-astronomer-certified.json'
    12        astroRuntimeReleasesFile: '/runtime-releases/local-astronomer-runtime.json'
    13        helm:
    14          airflow:
    15            extraEnv: |-
    16              - name: AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL
    17                value: "0"
    18    extraVolumes:
    19      - name: astronomer-runtime
    20        configMap:
    21          name: astronomer-runtime
    22          items:
    23            - key: astronomer-runtime.json
    24              path: local-astronomer-runtime.json
    25      - name: astronomer-certified
    26        configMap:
    27          name: astronomer-certified
    28          items:
    29            - key: astronomer-certified.json
    30              path: local-astronomer-certified.json
    31    volumeMounts:
    32      - name: astronomer-runtime
    33        mountPath: /runtime-releases
    34      - name: astronomer-certified
    35        mountPath: /certified-releases

Step 22: (OpenShift only) Apply OpenShift-specific configuration

If you’re not installing Astronomer Software into an OpenShift Kubernetes cluster, skip this step.

Add the following values into values.yaml. You can do this manually or by placing the configuration as a new file, along with merge_yaml.py in your project directory and running python merge_yaml.py openshift.yaml values.yaml.

1global:
2  openshiftEnabled: true
3  sccEnabled: false
4  extraAnnotations:
5    route.openshift.io/termination: "edge"
6  authSidecar:
7    enabled: true
8  dagOnlyDeployment:
9    securityContext:
10      fsGroup: ""
11  nodeExporterEnabled: false
12  fluentdEnabled: false
13  loggingSidecar:
14    enabled: true
15    name: sidecar-log-consumer
16elasticsearch:
17  sysctlInitContainer:
18    enabled: false
19
20# bundled postgresql not a supported option, only for use in proof-of-concepts
21postgresql:
22  securityContext:
23    enabled: false
24  volumePermissions:
25    enabled: false

Astronomer Software on OpenShift is only supported when using a third-party ingress-controller and using the logging sidecar feature of Astronomer Software. The above configuration enables both of these items.

Step 23: (Optional) Limit Astronomer to a namespace pool

By default, Astronomer Software automatically creates namespaces for each new Airflow Deployment.

You can restrict the Airflow management components of Astronomer Software to a list of predefined namespaces and configure it to operate without a ClusterRole by following the instructions in Configure a Kubernetes namespace pool for Astronomer Software. If you want to disable creation of role and rolebindings for commander, config-syncer, and kubestate metrics, you can set global.features.namespacePools.createRbac to false.

When global.rbacEnabled is set to false, the platform no longer creates any role, rolebindings, or service accounts. The user must define default roles to the k8s default service account to continue with the platform install. See Bring your own Kubernetes service accounts for setup steps.

Step 24: (Optional) Enable sidecar logging

Running a logging sidecar to export Airflow task logs is essential for running Astronomer Software in a multi-tenant cluster.

By default, Astronomer Software creates a privileged DaemonSet to aggregate logs from Airflow components for viewing from within Airflow and the Astronomer Software UI.

You can replace this privileged Daemonset with unprivileged logging sidecars by following instructions in Export logs using container sidecars.

Step 25: (Optional) Integrate an external identity provider

Astronomer Software includes integrations for several of the most popular OAUTH2 identity providers (IdPs), such as Okta and Microsoft Entra ID. Configuring an external IdP allows you to automatically provision and manage users in accordance with your organization’s security requirements. See Integrate an auth system to configure the identity provider of your choice in your values.yaml file.

Step 26: Create the load balancer

Skip this step if you’re using a third-party ingress controller or provisioning domain names for ingress objects using external DNS.

To install and function, Astronomer Software platform components require DNS entries to point to a load balancer associated with your ingress controller.

Perform a preliminary install of Astronomer Software to trigger your ingress controller to create the load balancer. This installation intentionally times out after 30 seconds, but it causes the ingress controller to still create the load balancer.

$./upgrade.sh --timeout 30s

Run kubectl -n <astronomer platformm namespace> get service -l component=ingress-controller and verify that a service-type LoadBalancer resource was created and that it received an External IP in a range that is accessible to your end-users, but not accessible to the general internet. For example:

$$kubectl -n astronomer get service -l component=ingress-controller
>NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
>astronomer-nginx           LoadBalancer   172.30.239.161   10.42.42.17     80:32697/TCP,443:30727/TCP   39m
>astronomer-nginx-metrics   ClusterIP      172.30.245.154   <none>          10254/TCP                    39m

If you uninstall and re-install the Astronomer Software platform chart, the ingress controller almost always receives a new IP address, which requires updating the DNS entries.

In lower environments, you can set nginx.loadBalancerIP in values.yaml to the External-IP address:

nginx:
  # IP address the nginx ingress should bind to
  loadBalancerIP: "10.42.42.17"

This option allows the cluster to request the same Load Balancer IP at creation time. Rebuilds immediately following teardowns can almost always receive the same IP address, but issuance is not guaranteed and installations will fail if the IP address has been assigned elsewhere, or is otherwise not available. Therefore, this option should not be used in higher environments unless you have taken special measures to guarantee the same IP address is reissued.

Step 27: Configure DNS for the integrated ingress controller

If you’re using a third-party ingress controller, you can skip this step.

The Astronomer load balancer routes incoming traffic to your NGINX ingress controller. After you install Astronomer Software, the load balancer will spin up in your cloud provider account.

Run $ kubectl get svc -n <astronomer platform namespace>to view your load balancer’s CNAME, located in the EXTERNAL-IP column for the astronomer-nginx service and look similar to the following:

1$ kubectl get svc -n astronomer
2NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                      AGE
3astronomer-alertmanager              ClusterIP      172.20.48.232    <none>                                                                    9093/TCP                                     24d
4[...]
5astronomer-nginx                     LoadBalancer   172.20.54.142    ELB_ADDRESS.us-east-1.elb.amazonaws.com                                   80:31925/TCP,443:32461/TCP,10254:32424/TCP   24d
6astronomer-nginx-default-backend     ClusterIP      172.20.186.254   <none>                                                                    8080/TCP                                     24d
7[...]

You will need to create a new CNAME record through your DNS provider using the external IP listed for for astronomer-nginx.

You can create a single wildcard CNAME record such as *.sandbox-astro.example.com, or alternatively create individual CNAME records for the following routes:

Astronomer Software requires the following domain-names be registered and resolvable within the Kubernetes cluster and to Astronomer Software users:

  • app.<base-domain> (required)
  • deployments.<base-domain> (required)
  • houston.<base-domain> (required)
  • prometheus.<base-domain> (required)
  • grafana.<base-domain> (required if using Astronomer Software’s integrated grafana)
  • kibana.<base-domain> (required if not using external elasticsearch)
  • registry.<base-domain> (required if using Astronomer Software’s integrated container-registry)
  • alertmanager.<base-domain> (required if using Astronomer Software’s integrated Alert Manager)
  • <base-domain> (optional but recommended, provides a vanity re-direct to app.<base-domain>)
  • install.<base-domain> (optional)

Step 28: Install Astronomer Software using Helm

Install the Astronomer Software Helm chart using upgrade.sh, which is recommended for your first install, or directly from Helm.

1./upgrade.sh --timeout 20m

Step 29: Verify Pods creation

To verify all pods are up and running, run:

1kubectl get pods --namespace <astronomer-platform-namespace>

All pods should be in Running status. For example,

1$ kubectl get pods --namespace astronomer
2
3NAME                                                       READY   STATUS              RESTARTS   AGE
4astronomer-alertmanager-0                                  1/1     Running             0          24m
5astronomer-astro-ui-7f94c9bbcc-7xntd                       1/1     Running             0          24m
6astronomer-astro-ui-7f94c9bbcc-lkn5b                       1/1     Running             0          24m
7<snip>

If all pods are not in running status, check the guide on debugging your installation or contact Astronomer support for additional configuration assistance.

Step 30: Verify you can access the Software UI

Visit https://app.<base-domain> in your web-browser to view Astronomer Software’s web interface.

Congratulations, you have configured and installed an Astronomer Software platform instance - your new Airflow control plane.

From the Astronomer Software UI, you’ll be able to both invite and manage users as well as create and monitor Airflow Deployments on the platform.

Additional information

The following topics include optional information about one or multiple topics in the installation guide.

Merge configurations with merge_yaml.py

When merging YAML configurations into values.yaml, you can merge manually or with a tool of your choosing.

You can use the followingmerge_yaml.py script to merge YAML excerpts into values.yaml automatically. This script requires both Python and the ruamel.yaml package, which you can install using pip install ruamel.yaml. To run the program, ensure that merge_yaml.py, values.yaml, and the yaml file that contains the configuration you want to add are all in your project directory. Then, run:

1python merge_yaml.py values-to-merge.yaml values.yaml
1#!/usr/bin/env python
2"""
3Backup destination file and merge YAML contents of src into dest.
4
5By default creates backups, overwrites destination, and clobbers lists.
6
7Usage:
8    merge_yaml.py src dest [--create-backup=True] [--dry-run] [--show-stacktrace=False] [--merge-lists=True] [--help]
9"""
10
11import argparse
12import os
13import shutil
14from datetime import datetime
15import sys
16from pathlib import Path
17
18# Check Python version
19if sys.version_info < (3, 0):
20    print("Error: This script requires Python 3.0 or greater.")
21    sys.exit(2)
22
23# Try importing ruamel.yaml
24try:
25    from ruamel.yaml import YAML
26except ImportError:
27    print(
28        "Error: ruamel.yaml is not installed. Please install it using 'pip install ruamel.yaml'"
29    )
30    sys.exit(2)
31
32yaml = YAML()
33
34def deep_merge(d1, d2, **kwargs):
35    """Deep merges dictionary d2 into dictionary d1."""
36    merge_lists = kwargs.get("merge_lists")
37    for key, value in d2.items():
38        if key in d1:
39            if isinstance(d1[key], dict) and isinstance(value, dict):
40                deep_merge(d1[key], value, **kwargs)
41            elif merge_lists and isinstance(d1[key], list) and isinstance(value, list):
42                d1[key].extend(value)
43            else:
44                d1[key] = value
45        else:
46            d1[key] = value
47    return d1
48
49def load_yaml_file(filename):
50    """Load YAML data from a file."""
51    if not os.path.exists(filename):
52        return {}
53    with open(filename, "r") as file:
54        return yaml.load(file)
55
56def save_yaml_file(filename, data):
57    """Save YAML data to a file."""
58    with open(filename, "w") as file:
59        yaml.dump(data, file)
60
61def create_backup(filename):
62    """Create a timestamped backup of the file."""
63    # create a directory called backups relative to the filename
64    backup_dir = filename.parent / "yaml_backups"
65    try:
66        backup_dir.mkdir(exist_ok=True)
67    except Exception as e:
68        print(
69            f"Error: Could not create backup directory {backup_dir}. Check your file-permissions or use --no-create-backup to skip creating a backup."
70        )
71        exit(2)
72
73    timestamp = datetime.now().strftime("%y%m%d%H%M%S")
74    backup_filename = backup_dir / f"{filename.name}.{timestamp}.bak"
75    shutil.copyfile(filename, backup_filename)
76    print(f"Backup created: {backup_filename}")
77
78def main():
79    parser = argparse.ArgumentParser(
80        description="Deep merge YAML contents of src into dest."
81    )
82    parser.add_argument("src", type=Path, help="Source filename")
83    parser.add_argument("dest", type=Path, help="Destination filename")
84    parser.add_argument(
85        "--create-backup",
86        type=bool,
87        default=True,
88        help="Create a backup of the destination file before merging",
89    )
90    parser.add_argument(
91        "--dry-run",
92        action="store_true",
93        help="Print to stdout only, do not write to the destination file",
94    )
95    # add a argument for showing the stack trace on yaml parse errors
96    parser.add_argument(
97        "--show-stacktrace",
98        action="store_true",
99        help="Show stack trace on yaml parse errors",
100    )
101    # add an argument to clobber lists
102    parser.add_argument(
103        "--merge-lists",
104        action="store_true",
105        help="Merge list items instead of clobbering",
106        default=False,
107    )
108
109    args = parser.parse_args()
110
111    src_filename = args.src.resolve().expanduser()
112    dest_filename = args.dest.resolve().expanduser()
113
114    # make sure both files exist
115    if not src_filename.exists():
116        print(f"Error: {args.src} does not exist")
117        exit(2)
118
119    if not dest_filename.exists():
120        print(f"Error: {args.dest} does not exist")
121        exit(2)
122
123    try:
124        src_data = load_yaml_file(src_filename)
125    except Exception as e:
126        print(
127            f"Error: {args.src} is not a valid YAML file. Run with --show-stacktrace to see the error."
128        )
129        if args.show_stacktrace:
130            raise e
131        exit(2)
132    try:
133        dest_data = load_yaml_file(dest_filename)
134    except Exception as e:
135        print(
136            f"Error: {args.dest} is not a valid YAML file. Run with --show-stacktrace to see the error."
137        )
138        if args.show_stacktrace:
139            raise e
140        exit(2)
141
142    if args.create_backup and not args.dry_run:
143        create_backup(dest_filename)
144
145    src_data = load_yaml_file(args.src)
146    dest_data = load_yaml_file(args.dest)
147
148    # if dest_data is empty, just copy src_data to dest_data
149    if not dest_data:
150        if not args.dry_run:
151            save_yaml_file(args.dest, src_data)
152    else:
153        merged_data = deep_merge(dest_data, src_data, merge_lists=args.merge_lists)
154        if not args.dry_run:
155            save_yaml_file(args.dest, merged_data)
156            print(f"Merged data from {args.src} into {args.dest}")
157        else:
158            yaml.dump(merged_data, sys.stdout)
159
160if __name__ == "__main__":
161    main()

Disable outbound email

You can configure Astronomer Software to not send outbound email.

Setting astronomer.houston.config.publicSignup: true with astronomer.houston.config.email.enabled: false is only secure when all non-OIDC authentication backends are explicitly disabled and the OIDC provider provides sufficient user validation to prevent untrusted users from accessing Astronomer Software.

To disable email transmission and email verification of users attempting to access the platform:

  1. In your values.yaml file, set astronomer.houston.config.email.enabled to false.
  2. Set astronomer.houston.config.publicSignups to true.
  3. Remove the EMAIL__SMTP_URL list-item from astronomer.houston.secret.

Configure Astronomer Software to trust private certificate authorities (CAs)

  1. Store the CA’s root public certificate to an Opaque Kubernetes secret in the Astronomer namespace with a descriptive name, such as private-root-ca, by running the following command.

Before you run this command, keep the following in mind:

  • The root certificate you specify should be the certificate of the authority that signed the Astronomer certificate. This is not the certificate associated with Astronomer or any other service.
  • The name of the secret file must be cert.pem for your certificate to be trusted properly.
  • The file must contain only a single certificate, it can’t be a certificate bundle.
1kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem

  1. Add <secret name> to the list of secret names contained in global.privateCaCerts in values.yaml:

    1global:
    2  privateCaCerts:
    3  - private-root-ca

Add trusted certificate authorities (CAs) to Docker Desktop

If your users will deploy images to a container registry, including the integrated container registry, that uses a TLS certificate signed by a private CA, you need to configure Docker Desktop to trust the CA’s public certificate.

Obtain a copy of the CA’s public certificate in pem format and place it in /etc/docker/certs.d:

1mkdir -p /etc/docker/certs.d
2cp privateCA.pem /etc/docker/certs.d/