Configure OpenLineage for a Remote Execution Agent

Airflow 3

This feature is only available for Airflow 3.x Deployments.

OpenLineage enables you to access data lineage and provenance across your Airflow workflows for your Remote Execution Agent. Features like Observe and Astro Alerts require that you enable OpenLineage for your data pipelines.

When you create your Remote Execution Agent, Astro automatically generates a Helm values.yaml file with OpenLineage configurations pre-filled. To set up OpenLineage, you need to configure an access credential for OpenLineage. This can be an Astro Deployment API token used as your OpenLineage API key. There are three methods you can use to add your API key to your Helm values:

Configure the API key as plain text: This stores your API key in your values.yaml file as plaintext, which is the simplest but least secure option. It would be appropriate for development or testing environments.
Use a pre-created Kubernetes secret: This procedure stores your API key separately from your values.yaml file, which provides more security than storing as plaintext. This option provides security with standard Kubernetes features.
Inject your API key with a secrets manager: This approach uses the init container to inject the Agent Token into the Remote Execution Agent component Pods. This example uses the Hashicorp Vault agent, but you can use your own secrets manager. This option provides enhanced security with the potential for secret rotation.

Prerequisites

A Kubernetes cluster
Helm
The values.yaml file downloaded from the Remote Execution Agent registration modal in the Astro UI
A Deployment API Token that uses a Custom Deployment role with Observe Ingest permissions

Setup

Step 1: Retrieve your OpenLineage Namespace and URL

In the Astro UI, go to the Deployment page and choose Agent. Then click Register Remote Agent.
Click Download values.yaml file.

The downloaded Helm values file will come with some of the key information for OpenLineage pre-filled, including OpenLineage Namespace and URL. So you only need to configure the OpenLineage API key.

Step 2: Configure the OpenLineage API Key

Configure key as plaintext

Use pre-created Kubernetes secret

Use secrets manager

This method stores an Astro Deployment API token as your OpenLineage API key as plain text in your values.yaml file, so that the Remote Execution Agent Helm chart can use it to creates a Kubernetes secret named openlineage-api-key-secret. This API key is base64-encoded in the Kubernetes secret.

All Remote Execution Agent components, such as the Worker, Dag processor, Triggerer, use this API key to authenticate with the OpenLineage endpoint.

Add the following OpenLineage configuration to your values.yaml file:

1 openlineage:
2   # Enable OpenLineage integration
3   enabled: true
4 
5   # Set your OpenLineage API key directly in the values file
6   apiKey: "insert-your-openlineage-api-key-here"
7 
8   # Do NOT set apiKeySecret when using apiKey
9 	# apiKeySecret: ~
10 
11   # Astro OpenLineage URL endpoint (This should be prefilled in the downloaded values.yaml from Astro UI)
12   url: "https://your-openlineage-endpoint.example.com"
13 
14   # Astro Deployment's namespace (This should be prefilled in the downloaded values.yaml from Astro UI)
15   namespace: "your-astro-deployment-namespace"

Apply the chart using the values.yaml file with the following command:

$ helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml