Install Astronomer Software on AWS EKS
This guide describes the steps to install Astronomer Software on Amazon Web Services (AWS), which allows you to deploy and scale Apache Airflow within an AWS Elastic Kubernetes Service (EKS) cluster.
Prerequisites
To install Astronomer on EKS, you’ll need access to the following tools and permissions:
- The AWS CLI.
- A compatible version of Kubernetes and PostgreSQL as described in the Astronomer Version Compatibility Reference.
- The Kubernetes CLI (kubectl).
- The OpenSSL CLI
- Helm (minimum v3.6).
- An SMTP service and credentials. For example, Mailgun or Sendgrid.
- Permission to create and modify resources on AWS.
- Permission to generate a certificate (not self-signed) that covers a defined set of subdomains.
- PostgreSQL superuser permissions.
- An AWS Load Balancer Controller for the IP target type is required for all private Network Load Balancers (NLBs). See Installing the AWS Load Balancer Controller add-on.
- If you use Kubernetes version 1.23 or later, the Amazon EBS CSI driver.
- Optional.
eksctl
for creating and managing your Astronomer cluster on EKS.
Step 1: Choose a base domain
All Astronomer services will be tied to a base domain of your choice, under which you will need the ability to add and edit DNS records.
Once created, your Astronomer base domain will be linked to a variety of sub-services that your users will access via the internet to manage, monitor and run Airflow on the platform.
For the base domain astro.mydomain.com
, for example, here are some corresponding URLs that your users would be able to reach:
- Software UI:
app.astro.mydomain.com
- Airflow Deployments:
deployments.astro.mydomain.com/deployment-release-name/airflow
- Grafana Dashboard:
grafana.astro.mydomain.com
- Kibana Dashboard:
kibana.astro.mydomain.com
For the full list of subdomains, see Step 4.
Step 2: Spin up the EKS control plane and a Kubernetes cluster
To proceed with the installation, you’ll need to spin up an EKS control plane as well as worker nodes in your Kubernetes cluster by following this AWS guide.
EKS is built off of Amazon’s pre-existing EC2 service, so you can manage your Kubernetes nodes the same way you would manage your EC2 nodes.
As you follow the guide linked above, keep in mind:
- Each version of Astronomer Software is compatible with only a particular set of Kubernetes versions. For more information, see the Astronomer Version Compatibility Reference.
- Astronomer recommends running the EKS control plane in a single security group. The worker nodes you spin up should have the same setup as the EKS control plane.
- All security and access settings needed for your worker nodes should be configured in your Cloud Formation template.
- Astronomer recommends configuring Autoscaling to automatically scale your resources up or down based on changing usage.
- If you create an EKS cluster from the UI,
kubectl
access will be limited to the user who created the cluster by default.- To give more users
kubectl
access, you’ll have to do so manually. - This post goes through how IAM plays with EKS.
- To give more users
- Expect to see each of your underlying nodes in the EC2 console.
- The default Astronomer resource requests are ~11 CPUs and ~40GiB of memory. Astronomer recommends using either six m5.xlarge or three m5.2xlarge instances for your cluster. To modify Astronomer default resource requests, see step 8.
kubectx
is an incredibly useful tool for quickly switching between Kubernetes clusters. Learn more here.Step 3: Create a namespace
Create a namespace called astronomer
to host the core Astronomer platform:
Once Astronomer is running, each Airflow Deployment that you create will have its own isolated namespace.
Step 4: Configure TLS
Astronomer recommends running Astronomer Software on a dedicated domain (BASEDOMAIN
) or subdomain (astro.BASEDOMAIN
).
In order for users to access the web applications they need to manage Astronomer, you’ll need a TLS certificate that covers the following subdomains:
To obtain a TLS certificate, complete one of the following setup options:
- Option 1: Obtain a TLS certificate from Let’s Encrypt. Astronomer recommends this option for smaller organizations where the DNS administrator and Kubernetes cluster administrator are the same person or on the same team.
- Option 2: Request a TLS certificate from your organization’s security team. Astronomer recommends this option for large organizations with their own protocols for generating TLS certificates.
- Option 3: Use the AWS Certificate Manager as the certificate provider.
Astronomer requires you to use RSA to sign TLS certificates. By default, Certbot uses Elliptic Curve Digital Signature Algorithm (ECDSA) keys to sign certificates. If you’re using Certbot to sign your TLS certificate, you must include -key-type rsa --rsa-key-size 2048
in your command to sign your certificate with an RSA key. If you don’t use RSA keys, deploys fail and error messages appear in the registry and Houston logs. For example, you can run the following command to sign your certificate with an RSA key:
Option 1: Create TLS certificates using Let’s Encrypt
Let’s Encrypt is a free and secure certificate authority (CA) service that provides TLS certificates that renew automatically every 90 days. Use this option if you are configuring Astronomer for a smaller organization without a dedicated security team.
To set up TLS certificates this way, follow the guidelines in Automatically Renew TLS Certificates Using Let’s Encrypt. Make note of the certificate you create in this setup for Step 5.
Option 2: Request a TLS certificate from your security team
If you’re installing Astronomer for a large organization, you’ll need to request a TLS certificate and private key from your enterprise security team. This certificate needs to be valid for the BASEDOMAIN
your organization uses for Astronomer, as well as the subdomains listed at the beginning of Step 4. You should be given two .pem
files:
- One for your encrypted certificate
- One for your private key
To confirm that your enterprise security team generated the correct certificate, run the following command using the openssl
CLI:
This command will generate a report. If the X509v3 Subject Alternative Name
section of this report includes either a single *.BASEDOMAIN
wildcard domain or the subdomains listed at the beginning of Step 4, then the certificate creation was successful.
Depending on your organization, you may receive either a globally trusted certificate or a certificate from a private CA. The certificate from your private CA may include a domain certificate, a root certificate, and/or intermediate certificates, all of which need to be in proper certificate order. To verify certificate order, follow the guidelines below.
Option 3: Use the AWS Certificate Manager as the certificate provider
AWS Certificate Manager (ACM) terminates TLS at the Load Balancer level and does not encrypt the internal traffic inside the cluster. The ACM ingress controller requires an astronomer-tls
secret with a certificate and private key to encrypt internal traffic.
Because ACM relies on annotations attached to Kubernetes resources and does not issue certificates or private key files, you must complete one of the following options to create the astronomer-tls
secret:
- Create a self-signed certificate. Self-signed certificates are ideal for privately hosted internal applications, as well as in development and testing environments. Avoid using self-signed certificates in installations where the trust and identity of the certificate issuer are important.
- Create a certificate with AWS Certificate Manager Private CA.
- Create a certificate with Let’s Encrypt.
After you create the certificate, add the following configuration block to your Helm chart in Step 8, making sure to replace the <ACM-Certificate-ARN>
field with the ARN of your ACM certificate.
Confirm certificate chain order
If your organization is using a private certificate authority, you’ll need to confirm that your certificate chain is ordered correctly. To determine your certificate chain order, run the following command using the openssl
CLI:
The command generates a report of all certificates. Verify the order of the certificates is as follows:
- Domain
- Intermediate (optional)
- Root
If the certificate order is correct, proceed to step 5.
Step 5: Create a Kubernetes TLS Secret
If you received a globally trusted certificate or created a self-signed certificate, create a Kubernetes TLS secret using the following command and then proceed to Step 6:
If you created a certificate using Let’s Encrypt, the astronomer-tls
secret already exists in your Kubernetes cluster. Run the following command to confirm it exists:
If you received a certificate from a private CA, follow these steps instead:
-
Add the root certificate provided by your security team to an Opaque Kubernetes secret in the Astronomer namespace by running the following command:
cert.pem
for your certificate to be trusted properly.- Note the value of
private-root-ca
for when you configure your Helm chart in Step 8. You’ll need to additionally specify theprivateCaCerts
key-value pair with this value for that step.
Step 6: Configure your SMTP URI
An SMTP service is required for sending and accepting email invites from Astronomer. If you’re running Astronomer Software with publicSignups
disabled (which is the default), you’ll need to configure SMTP as a way for your users to receive and accept invites to the platform via email. To integrate your SMTP service with Astronomer, fetch your SMTP service’s URI and store it in a Kubernetes secret:
In general, an SMTP URI will take the following form:
The following table contains examples of what the URI will look like for some of the most popular SMTP services:
If your SMTP provider is not listed, refer to the provider’s documentation for information on creating an SMTP URI.
/
or other escape characters in your username or password, you may need to URL encode those characters.Step 7: Configure the database
By default, Astronomer requires a central Postgres database that will act as the backend for Astronomer’s Houston API and will host individual metadata databases for all Airflow Deployments spun up on the platform.
While you’re free to configure any database, most AWS users on Astronomer run Amazon RDS for PostgreSQL. For production environments, Astronomer recommends a managed Postgres solution.
If you’re setting up a development environment, this step is optional. Astronomer can be configured to deploy the PostgreSQL helm chart as the backend database with the following set in your values.yaml
:
To connect to an external database to your EKS cluster, create a Kubernetes Secret named astronomer-bootstrap
that points to your database.
Step 8: Configure your Helm chart
As a next step, create a file named values.yaml
in an empty directory.
For context, this values.yaml
file will assume a set of default values for our platform that specify everything from user role definitions to the Airflow images you want to support. As you grow with Astronomer and want to customize the platform to better suit your team and use case, your values.yaml
file is the best place to do so.
Copy and paste the following example into the values.yaml
file. Replace baseDomain
, private-root-ca
, /etc/containerd/certs.d
, astronomer.houston.secret
, and ssl.enabled
with your own values. Additional example configurations are available in the Astronomer GitHub configs repository.
These are the minimum values you need to configure for installing Astronomer. For information on additional configuration, read What’s Next.
Step 9: Install Astronomer
Now that you have an EKS cluster set up and your values.yaml
file defined, you’re ready to deploy all components of our platform.
First, run:
Then, run:
This ensures that you pull the latest image from the Astronomer Helm repository. Now, run:
This command installs the most recent patch version of Astronomer Software. To install a different patch version, add the --version=
flag and use the format 0.34.x
. For example, to install Astronomer Software v0.34.0, you specify --version=0.34.0
. For more information about the available patch versions, see the Software Release Notes.
When you’re defining <your-platform-release-name>
, Astronomer recommends limiting the name to 12 characters to avoid operational issues.
After you run the previous commands, a set of Kubernetes pods are generated in your namespace. These pods power the individual services required to run the Astronomer platform, including the Software UI and Houston API.
Alternative ArgoCD installation
You can install Astronomer with ArgoCD, which is an open source continuous delivery tool for Kubernetes, as an alternative to using helm install
.
Because ArgoCD doesn’t support sync wave dependencies for app of apps structures, installing Astronomer requires some additional steps compared to the standard ArgoCD workflow:
-
Under the
global
section of yourvalues.yaml
file, addenableArgoCDAnnotation: true
. -
Create a new ArgoCD app. When creating the app, configure the following:
- Path: The filepath of your
values.yaml
file - Namespace: The namespace you want to use for Astronomer
- Cluster: The Kubernetes cluster in which you’re installing Astronomer
- Repository URL:
https://helm.astronomer.io
- Path: The filepath of your
-
Sync the ArgoCD app with every component of the Astronomer platform selected. See Sync (Deploy) the Application.
-
Stop the sync when you see that
astronomer-houston-db-migrations
has completed in the Argo UI. -
Sync the application a second time, but this time clear
astronomer-alertmanager
in the Argo UI while keeping all other components selected. Wait for this sync to finish completely. -
Sync the ArgoCD app a third time with all Astronomer platform components selected.
Step 10: Verify Pods are up
To verify all pods are up and running, run:
You should see something like this:
If you are seeing issues here, check out our guide on debugging your installation.
Step 11: Configure DNS
Now that you’ve successfully installed Astronomer, a new load balancer will have spun up in your AWS account. This load balancer routes incoming traffic to our NGINX ingress controller.
Run $ kubectl get svc -n astronomer
to view your load balancer’s CNAME, located under the EXTERNAL-IP
column for the astronomer-nginx
service.
You will need to create a new CNAME record through your DNS provider using the ELB CNAME listed above. If you’re using Amazon Route 53, see Creating records by using the Amazon Route 53 console.
You can create a single wildcard CNAME record such as *.astro.mydomain.com
, or alternatively create individual CNAME records for the following routes:
Example wildcard CNAME record:
Step 12: Verify you can access the Software UI
Go to app.BASEDOMAIN
to see the Software UI.
Consider this your new Airflow control plane. From the Software UI, you’ll be able to both invite and manage users as well as create and monitor Airflow Deployments on the platform.
Step 13: Verify your TLS setup
To check if your TLS certificates were accepted, log in to the Software UI. Then, go to app.BASEDOMAIN/token
and run:
Verify that this output matches with that of the following command, which doesn’t look for TLS:
Next, to make sure the registry is accepted by Astronomer’s local docker client, try authenticating to Astronomer with the Astro CLI:
If you can log in, then your Docker client trusts the registry. If Docker does not trust the Astronomer registry, run the following and restart Docker:
Finally, try running $ astro deploy
on a test deployment. Create a deployment in the Software UI, then run:
Check the Airflow namespace. If pods are changing at all, then the Houston API trusts the registry.
If you have Airflow pods in the state ImagePullBackoff
, check the pod description. If you see an x509 error, ensure that you have:
- Configured containerd’s
config_path
to point to/etc/containerd/certs.d
. - Added the
privateCaCertsAddToHost
key-value pairs to your Helm chart.
If you missed these steps during installation, follow the steps in Apply a config change to add them after installation. If you are using a base image such as CoreOS that does not permit values to be changed, or you otherwise can’t modify values.yaml
, contact Astronomer support for additional configuration assistance.
What’s next
To help you make the most of Astronomer Software, check out the following additional resources:
- Renew TLS Certificates on Astronomer Software
- Integrating an Auth System
- Configuring Platform Resources
- Managing Users on Astronomer Software
Astronomer support team
If you have any feedback or need help during this process and aren’t in touch with our team already, a few resources to keep in mind:
- Community Forum: Search previously asked questions about Airflow + Astronomer FAQs
- Astronomer Support Portal: Platform or Airflow issues
For detailed guidelines on reaching out to Astronomer Support, reference our guide here.