Configure AWS PrivateLink for Remote Execution Agents
Airflow 3
This feature is only available for Airflow 3.x Deployments.AWS PrivateLink enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a VPC Endpoint in your AWS environment to establish secure communication with Astro.
Overview
By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With AWS PrivateLink, you can route this traffic through a private connection within AWS, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.
The setup involves creating a VPC Endpoint in your AWS account that connects to Astronomer’s VPC Endpoint Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.
Prerequisites
- An Astro Deployment configured for Remote Execution.
- Remote Execution Agents installed in an AWS environment.
- Access to the AWS Console with permissions to create VPC Endpoints and modify Route53 configurations.
Astro-side configuration
Before you can create a VPC Endpoint, Astronomer must configure the VPC Endpoint Service on the Astro side. Contact Astronomer Support with the following information:
- Your Astro Cluster ID.
- The AWS Account ID where your Remote Execution Agents are running.
- The AWS Region where your Remote Execution Agents are running.
Astronomer Support will provide you with the VPC Endpoint Service name, Service region, and supported Availability Zones required to create your VPC Endpoint.
If your Remote Execution Agents run in a different AWS region than the Astro orchestration plane, inform Astronomer Support. Additional configuration may be required on the Astro side, such as adding your region to the VPC Endpoint Service cross-region configuration or adding your AWS account to the allowed principals list.
Create a VPC Endpoint
After receiving the VPC Endpoint Service name from Astronomer Support, create a VPC Endpoint in your AWS account.
Configure the endpoint
Set the following values:
- Name tag: Enter a descriptive name, such as
astro-privatelink. - Type: Select Endpoint services that use NLBs and GWLBs.
- Service name: Enter the VPC Endpoint Service name provided by Astronomer Support, and click Verify service to confirm the service name is valid.
- Cross-Region: Enable if required (optional).
- VPC: Select the VPC where your Remote Execution Agents are running.
- Subnets: Select at least one subnet. For high availability, select subnets in multiple Availability Zones.
- Security group: Select or create a security group that allows inbound traffic on HTTPS port 443.
Subnet selection shows your subnets in the Availability zones supported by the VPCe Service. If there is a mismatch, you must create subnet(s) in the zones provided by Astronomer Support.
Configure DNS resolution
After creating the VPC Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the private endpoint IP addresses.
Configure Route53 private hosted zone
Verify the connection
After completing the configuration, verify that your Remote Execution Agents can communicate with Astro through the private endpoint. Validate in the Astro UI that the agents are heart beating and reporting a Healthy status. You can also verify from within your network using the below instructions.
- Connect to a host within your VPC that has network access to the VPC Endpoint.
- Run a DNS lookup to confirm the hostname resolves to a private IP address:
The response should show the private IP addresses assigned to your VPC Endpoint rather than public IP addresses.
- Test connectivity to the endpoint:
The expected response is 404 page not found. If the connection is successful, your Remote Execution Agents will use the private endpoint for all communication with the Astro orchestration plane.
Multiple Remote Execution Agents
If you have multiple Remote Execution Agents across different VPCs, you can either create a VPC Endpoint in each VPC, or use a single VPC Endpoint and configure network routing between VPCs.
The following table summarizes the actions required based on your configuration:
If you previously created a Route53 private hosted zone, you can associate additional VPCs with the same hosted zone rather than creating new zones for each VPC.
Restrict traffic to the private endpoint
After verifying that the private endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the VPC Endpoint. This ensures that all communication with Astro uses the private connection.
To restrict traffic:
- Take note of your Astro Cluster ID, under Organization Settings > Clusters > Cluster details.
- In the Astro UI, navigate to your Deployment and go to Settings.
- In your Deployment Advanced settings, add the cluster CIDR range to the Allowed IP address ranges list.
This configuration ensures that only traffic coming through the VPC Endpoint Service can reach the Deployment.
Troubleshooting
VPC Endpoint shows “pending acceptance”
The VPC Endpoint Service may require manual acceptance of endpoint connections, if still in pending state after 5 minutes. Contact Astronomer Support to approve your endpoint connection request.
DNS resolution returns public IP addresses
Verify that your Route53 private hosted zone is correctly configured and associated with the VPC where you are testing.
Connection timeouts
Check that the security group attached to the VPC Endpoint allows inbound traffic on port 443 from the subnets where your Remote Execution Agents are running.
