Create a network connection between Astro and AWS
You can grant Astro cluster and its Deployments access to your external AWS resources.
Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to AWS through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an AWS resource’s allowlist.
If you have stricter security requirements, you can create a private connection to AWS in a few different ways.
After you create a connection from your cluster to AWS, you might also need to individually authorize Deployments to access specific resources. See Authorize your Deployment using workload identity.
Standard and dedicated cluster support for AWS networking
Standard clusters have different connection options than dedicated clusters.
Standard clusters can connect to AWS in the following ways:
- Using static external IP addresses
- Using PrivateLink to connect with the following endpoints:
- Amazon S3 - Gateway Endpoint
- Amazon Simple Queue Service (SQS) - Interface Endpoint - Amazon Elastic Container Registry (ECR) - Interface Endpoints for ECR API and Docker Registry API
- Elastic Load Balancing (ELB) - Interface Endpoint
- AWS Security Token Service (AWS STS) - Interface Endpoint
Dedicated clusters can connect to AWS in the same ways as standard clusters. Additionally, they support a number of private connectivity options including:
- VPC peering
- Transit Gateways
- AWS PrivateLink
If you require a private connection between Astro and AWS, Astronomer recommends configuring a dedicated cluster. See Create a dedicated cluster. Transitive connectivity to on-premise networks is also possible through your managed VPCs. However, architectures with a demarcation point between Astro and your on-premise network are not supported.
Access a public AWS endpoint
All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns the external IP addresses to it. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your AWS resources through a valid Airflow connection.
Allowlist a Deployment’s external IP addresses on AWS
- In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
- Select the Details tab.
- In the Other section, you can find the External IPs associated with the Deployment.
- Add the IP addresses to the allowlist of any external services that you want your Deployment to access.
When you use publicly accessible endpoints to connect to AWS, traffic moves directly between your Astro cluster and the AWS API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can Authorize deployments to your cloud with workload identity so that you can avoid adding passwords or other access credentials to your Airflow connections.
Dedicated cluster external IP addresses
If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list of cluster-level external IP addresses in the Clusters page of the Astro UI.
- In the Organization section of the Astro UI, click Organization Settings, then click Clusters, then select a cluster.
- In the Details page, copy the IP addresses listed under External IPs.
- Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the Details page of any Deployment in the cluster.
After you allowlist a cluster’s IP addresses, all Deployments in that cluster have network connectivity to AWS.