Configure Azure Private Link for Remote Execution Agents

Airflow 3

This feature is only available for Airflow 3.x Deployments.

Azure Private Link enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a Private Endpoint in your Azure environment to establish secure communication with Astro.

Overview

By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With Azure Private Link, you can route this traffic through a private connection within Azure, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.

The setup involves creating a Private Endpoint in your Azure subscription that connects to Astronomer’s Private Link Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.

Prerequisites

An Astro Deployment configured for Remote Execution.
Remote Execution Agents installed in an Azure environment (Azure Kubernetes Service (AKS)).
Access to the Azure portal with permissions to create Private Endpoints, Private DNS zones, and modify networking configurations.

Astro-side configuration

Before you can create a Private Endpoint, Astronomer must configure the Private Link Service on the Astro side. Contact Astronomer support with the following information:

Your Astro Cluster ID.

Astronomer support will provide you with the Private Link Service alias required to create your Private Endpoint.

Create a Private Endpoint

After receiving the Private Link Service alias from Astronomer Support, create a Private Endpoint in your Azure subscription.

Gather required information

Before starting, collect the following:

The Private Link Service alias provided by Astronomer support.
The Remote Execution API URL for your Deployment, which you can find in the Deployment details in the Astro UI.

Navigate to Private Endpoints

In the Azure portal, go to Private Link Overview > Private endpoints.

Start the endpoint creation wizard

Click Create to begin the configuration.

Configure the endpoint

Set the following values:

Resource group: Select the resource group where your AKS cluster with Remote Execution Agents is deployed.
Name: Enter a descriptive name, such as astro-privatelink.
Network Interface Name: Accept the default or enter a custom name.
Region: Select the region where your Remote Execution Agents are running.

Configure the resource connection

Set the following values:

Connection method: Select Connect to an Azure resource by resource ID or alias.
Resource ID or alias: Enter the Private Link Service alias provided by Astronomer support.

Configure networking

Set the following values:

Virtual Network: Select the VNet where your AKS cluster with Remote Execution Agents is deployed.
Subnet: Select the appropriate subnet based on your internal network preferences.

Create the endpoint

Review your configuration and click Create.

After creating the Private Endpoint, contact Astronomer support to approve your endpoint connection request. You can proceed with the DNS configuration while waiting for approval.

Configure DNS resolution

After creating the Private Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the Private Endpoint IP address.

Create a private DNS zone

Navigate to Private DNS zones

In the Azure portal, go to Private DNS zones.

Create a new zone

Click Create.
Enter external.astronomer.run as the zone name.
Select the resource group and click Create.

Link the DNS zone to your VNet

In the newly created DNS zone, go to Virtual network links.
Click Add.
Enter a link name and select the VNet where your AKS cluster is deployed.
Click OK.

Create an A record

In the DNS zone, click Record set.
For Name, enter the first subdomain from your Remote Execution API URL. For example, if your API URL is clxxxxxxxxx.external.astronomer.run, enter clxxxxxxxxx.
For Type, select A.
For IP address, enter the Private Endpoint IP address. You can find this in the Private Endpoint’s Network interface settings in the Azure portal.
Click OK.

Verify the connection

After Astronomer support approves your endpoint connection, verify that your Remote Execution Agents can communicate with Astro through the Private Endpoint. Validate in the Astro UI that the agents are heartbeating and reporting a Healthy status. You can also verify from within your network using the following instructions.

Connect to a host within your VNet that has network access to the Private Endpoint.
Run a DNS lookup to confirm the hostname resolves to a private IP address:

$ nslookup <AstroClusterId>.external.astronomer.run

The response should show the private IP address assigned to your Private Endpoint rather than a public IP address.

Test connectivity to the endpoint:

$ curl -v https://<AstroClusterId>.external.astronomer.run

The expected response is 404 page not found. If the connection is successful, your Remote Execution Agents will use the Private Endpoint for all communication with the Astro orchestration plane.

Multiple Remote Execution Agents

Only one Private Link Service is required per Astro cluster. If you have multiple Remote Execution Agents across different VNets, you can either create a Private Endpoint in each VNet (the Private Link Service alias remains unchanged) or use a single Private Endpoint across your network.

If you previously created a private DNS zone, you can associate additional VNets with the same zone rather than creating new zones for each VNet.

Restrict traffic to the Private Endpoint

After verifying that the Private Endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the Private Endpoint. This ensures that all communication with Astro uses the private connection.

To restrict traffic:

Take note of your Astro Cluster ID, under Organization Settings > Clusters > Cluster details.
In the Astro UI, navigate to your Deployment and go to Settings.
In your Deployment Advanced settings, add the cluster CIDR range to the Allowed IP address ranges list.

This configuration ensures that only traffic coming through the Private Link Service can reach the Deployment.

Troubleshooting

Private Endpoint shows “pending” connection state

The Private Link Service requires manual acceptance of endpoint connections. Contact Astronomer support to approve your endpoint connection request.

DNS resolution returns public IP addresses

Verify that your private DNS zone is correctly configured and linked to the VNet where you are testing. Ensure the A record points to the correct Private Endpoint IP address.

Connection timeouts

Check that your network security group (NSG) rules allow outbound traffic on port 443 from the subnets where your Remote Execution Agents are running to the Private Endpoint.