Skip to main content

Create a network connection between Astro and AWS

Use this document to learn how you can grant Astro cluster and its Deployments access to your external AWS resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to AWS through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an AWS resource's allowlist.

If you have stricter security requirements, you can create a private connection to AWS in a few different ways.

After you create a connection from your cluster to AWS, you might also need to individually authorize Deployments to access specific resources. See Authorize your Deployment using workload identity.

Standard and dedicated cluster support for AWS networking​

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to AWS in the following ways:

Dedicated clusters can connect to AWS in the same ways as standard clusters. Additionally, they support a number of private connectivity options including:

  • VPC peering
  • Transit Gateways
  • AWS PrivateLink

If you require a private connection between Astro and AWS, Astronomer recommends configuring a dedicated cluster. See Create a dedicated cluster. Transitive connectivity to on-premise networks is also possible through your managed VPCs. However, architectures with a demarcation point between Astro and your on-premise network are not supported.

Access a public AWS endpoint​

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns the external IP addresses to it. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your AWS resources through a valid Airflow connection.

Allowlist a Deployment's external IP addresses on AWS​

  1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
  2. Select the Details tab.
  3. In the Other section, you can find the External IPs associated with the Deployment.
  4. Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to AWS, traffic moves directly between your Astro cluster and the AWS API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can Authorize deployments to your cloud with workload identity so that you can avoid adding passwords or other access credentials to your Airflow connections.

Dedicated cluster external IP addresses

If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list of cluster-level external IP addresses in the Clusters page of the Astro UI.

  1. In the Organization section of the Astro UI, click Organization Settings, then click Clusters, then select a cluster.
  2. In the Details page, copy the IP addresses listed under External IPs.
  3. Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the Details page of any Deployment in the cluster.

After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to AWS.

Create a private connection between Astro and AWS​

Choose one of the following setups based on the security requirements of your company and your existing infrastructure.

info

This connection option is only available for dedicated Astro clusters.

warning

Self-service VPC configuration on Astro is in Public Preview.

Prerequisites​

  • An external VPC on AWS
  • A CIDR block for your external VPC in the RFC 1918 range
  • Organization Owner permissions

Setup​

To set up a private connection between an Astro VPC and an AWS VPC, you can create a VPC peering connection. VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts.

  1. Open the AWS console of the AWS account with the external VPC and copy the following:

    • AWS account ID
    • AWS region
    • VPC ID of the external VPC
    • CIDR block of the external VPC

  2. In the Astro UI, click Organization Settings, then click Clusters, select your cluster, click VPC Peering Connections, then click + VPC Peering Connection.

  3. Configure the following values for your VPC peering connection using the information you copied in Step 1:

    • Peering Name: Provide a name for the VPC peering connection.
    • AWS account ID: Enter the account ID of the external VPC.
    • Destination VPC ID: Enter the VPC ID.
    • Destination VPC region: Enter the region of the external VPC.
    • Destination VPC CIDR block: Enter the CIDR block of the external VPC.

  4. Click Create Connection. The connection appears as Pending.

  5. Wait a few minutes for the Complete Activation button to appear, then click Complete Activation link.

  6. In the modal that appears, follow the instructions to accept the connection from your external VPC and create routes from the external VPC to Astro.

A few minutes after you complete the instructions in the modal, the connection status changes from Pending to Active. A new default route appears in Routes with your configured CIDR block.

Troubleshooting VPC connection statuses

Astro might show additional information in your connection status if it has an issue when it creates the connection. The following are all possible connection statuses.

  • Pending (Without Complete Activation): Astro is sending the peering request to the external VPC. Wait 1-2 minutes for request to be created and sent.
  • Pending (With Complete Activation): The peering connection request has been created and sent. Click Complete Activation to finish the setup.
  • Active: The peering connection was successfully created and accepted.
  • Failed: The peering connection request was rejected. Delete the failed connection and retry using a new connection configuration. If you don't delete the failed connection, Astro will retry creating the peering request whenever you create a new VPC connection.
  • Not Found: Astro failed to create the peering request. Wait 5 minutes for Astro to retry. If the status hasn't changed after 5 minutes, delete the connection and retry using a new connection configuration.

Note that a VPC connection can be listed as Active even when it has an incorrectly configured CIDR block. To reconfigure your CIDR block without deleting your connection, delete the route that was generated when you configured the connection and create a new route with the correct CIDR block.

Configure additional routes for a VPC connection​

Your initial VPC connection connects Astro to your external VPC through a primary CIDR block. To connect Astro to other data services or systems within the external VPC, you can create additional routes to secondary CIDR blocks or subnets within the primary CIDR block. You can also complete this setup if you recently configured a new service in your external VPC and want to connect it with Astro without updating your base VPC connection.

  1. Open the Routes tab, then click + Route.

  2. Configure the following details for your route:

    • Route ID: Provide a name for the route.
    • Destination: Enter the subnet of the service in the external VPC.
    • Target: Select the VPC peering connection you configured.

  3. Click Create Route, then wait a few minutes for the route to be created.

DNS considerations for VPC peering​

If your external VPC resolves DNS hostnames using DNS Hostnames and DNS Resolution, you must also enable the Accepter DNS Resolution setting on AWS. This allows Astro clusters and Deployments to resolve the public DNS hostnames of the external VPC to its private IP addresses. To configure this option, see AWS Documentation.

If your external VPC resolves DNS hostnames using private hosted zones, then you must associate your Route53 private hosted zone with the Astro VPC using instructions provided in AWS Documentation.

To retrieve the ID of any Astro VPC, contact Astronomer support. If you have more than one Astro cluster, request the VPC ID of each cluster.

Hostname resolution options​

Securely connect Astro to resources running in other VPCs or on-premises through a resolving service.

Using Route 53 requires sharing a resolver rule with your Astro account. If this is a security concern, Astronomer recommends using Domain Name System (DNS) forwarding.If you have a small number of records and immutable IP addresses, the Astronomer support team can create a Private zone with DNS records, pointed to your resources.

Use Route 53 Resolver rules to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises.

Prerequisites​

  • An Amazon Route 53 Resolver rule. See Managing forwarding rules.
  • Permission to share resources using the AWS Resource Access Manager (RAM)

Share the Amazon Route 53 Resolver rule​

To allow Astro to access a private hosted zone, you need to share your Amazon Route 53 Resolver rule with your Astro AWS account.

  1. In the Route 53 Dashboard, click Rules below Resolver in the navigation menu.

  2. Select a Resolver rule and then click Details.

  3. Click Share and enter Astro in the Name field.

  4. In the Resources - optional section, select Resolver Rules in the Select resource type list and then select one or more rules.

  5. On the Associate permissions page, accept the default settings and then click Next.

  6. On the Grant access to principals page, select Allow sharing only within your organization, and then enter your Astro AWS account ID for your organization in the Enter an AWS account ID field.

    To get the Astro AWS account ID, in the Astro UI, click Organization Settings. From the General page, copy the AWS External ID.

  7. Click Create resource share.

Contact Astronomer support for rule verification​

To verify that the Amazon Route 53 Resolver rule was shared correctly, submit a request to Astronomer support. With your request, include the Amazon Route 53 Resolver rule ID. To locate the Resolver rule ID, open the Route 53 Dashboard, and in the left menu click Rules below Resolver. Copy the value in the Resolver ID column.

Create a connection to confirm connectivity (optional)​

When Astronomer support confirms that the Amazon Route 53 Resolver rule was successfully associated with the Astro VPC, you can create a connection to the resource that is resolved by the shared rule. See Managing Connections.

See Also​

Was this page helpful?