> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure OpenLineage for a Remote Execution Agent

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

OpenLineage enables you to access data lineage and provenance across your Airflow workflows for your Remote Execution Agent. Features like [Observe](/docs/astro/astro-observe) and [Astro Alerts](/docs/astro/alerts) require that you enable OpenLineage for your data pipelines.

When you create your Remote Execution Agent, Astro automatically generates a Helm `values.yaml` file with OpenLineage configurations pre-filled. To set up OpenLineage, you need to configure an access credential for OpenLineage. This can be an [Astro Deployment API token](/docs/astro/deployment-api-tokens) used as your OpenLineage API key. There are three methods you can use to add your API key to your Helm values:

* Method 1 - Configure the API key as plain text. This stores your API key in your `values.yaml` file as plaintext, which is the simplest but least secure option. It's appropriate for development or testing environments.
* Method 2 - Use a pre-created Kubernetes secret. This procedure stores your API key separately from your `values.yaml` file, which provides more security than storing as plaintext. This option provides security with standard Kubernetes features.
* Method 3 - Inject your API key with a secrets manager. This approach uses an init container to inject the Agent token into the Remote Execution Agent component Pods. This example uses the [HashiCorp Vault](https://developer.hashicorp.com/vault) Agent, but you can use your own secrets manager. This option provides enhanced security with the potential for secret rotation.

## Prerequisites

* A Kubernetes cluster
* [Helm](https://helm.sh/docs/helm/helm_install/)
* The `values.yaml` file downloaded from the Remote Execution Agent registration modal in the [Astro UI](https://cloud.astronomer.io)
* A [Deployment API Token](/docs/astro/deployment-api-tokens) that uses a [Custom Deployment role](/docs/astro/customize-deployment-roles) with **Observe Ingest** permissions

## Setup

### Step 1: Retrieve your OpenLineage variables

<Steps>
  <Step title="Open the Remote Agent registration">
    In the [Astro UI](https://cloud.astronomer.io), go to the **Deployment** page and choose **Agent**. Then click **Register Remote Agent**.

    #### Download the values file

    Click **Download `values.yaml` file**.
  </Step>
</Steps>

The downloaded Helm values file includes most OpenLineage variables pre-filled, so you only need to configure the OpenLineage API key.

### Step 2: Configure the OpenLineage API key

<Tabs>
  <Tab title="Configure key as plaintext">
    This method stores an Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key in plain text in your `values.yaml` file, so that the Remote Execution Agent Helm chart can use it to create a Kubernetes secret named `openlineage-api-key-secret`. This API key is base64-encoded in the Kubernetes secret.

    All Remote Execution Agent components — the worker, Dag processor, and triggerer — use this API key to authenticate with the OpenLineage endpoint.

    <Steps>
      <Step title="Update your values file">
        Add the following OpenLineage configuration to your `values.yaml` file:

        ```yaml title="values.yaml" wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Set your OpenLineage API key directly in the values file
          apiKey: "<OPENLINEAGE_API_KEY>"

          # Do NOT set apiKeySecret when using apiKey
          # apiKeySecret: ~

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'
        ```

        #### Install the Helm chart

        Apply the chart using the `values.yaml` file with the following command:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Use pre-created Kubernetes secret">
    When you use this method, the Remote Execution Agent Helm chart doesn't create a new secret for the OpenLineage API key. Instead, it configures all Agent components — the worker, Dag processor, and triggerer — to use the existing secret to authenticate with the OpenLineage endpoint.

    The secret must have a key named `api-key` containing the OpenLineage API key. You can use an Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key.

    <Steps>
      <Step title="Create the Kubernetes secret">
        Create a Kubernetes secret containing your OpenLineage API key:

        ```sh wrap theme={null}
        kubectl create secret generic openlineage-api-key-secret \
          --from-literal=api-key=<OPENLINEAGE_API_KEY> \
          --namespace <YOUR_NAMESPACE>
        ```

        #### Reference the secret from your values file

        Configure your `values.yaml` file so that OpenLineage uses your pre-created secret:

        ```yaml title="values.yaml" wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Do NOT set apiKey when using apiKeySecret
          # apiKey: ~

          # Reference the pre-created secret you created in the previous step
          apiKeySecret: "openlineage-api-key-secret"

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'
        ```

        #### Install the Helm chart with the referenced secret

        Apply the chart using the `values.yaml` file with the following command:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Use secrets manager">
    You can also use a secrets manager to securely store your API keys. The following procedure specifically uses the [HashiCorp Vault Agent](https://developer.hashicorp.com/vault/tutorials/vault-agent).

    The Vault Agent init container runs before the main Remote Execution Agent container. The Vault Agent authenticates with Vault to retrieve the OpenLineage API key and then writes the API key to a file in the shared volume. The Remote Execution Agent container can then read the OpenLineage API key from the file using the `OPENLINEAGE_API_KEY` environment variable. Use your Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key.

    <Steps>
      <Step title="Configure OpenLineage and init containers">
        Configure OpenLineage and add init containers for the `dagProcessor`, `workers`, and `triggerer` components:

        ```yaml title="values.yaml" expandable wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Don't set apiKey or apiKeySecret
          # apiKey: ~
          # apiKeySecret: ~

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'

        # Configure each component to use the Vault init container
        dagProcessor:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key

        triggerer:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key

        workers:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key
        ```

        #### Create a ConfigMap for the Vault Agent

        ```sh wrap theme={null}
        cat <<EOF | kubectl apply -f -
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: vault-agent-config
          namespace: <YOUR_NAMESPACE>
        data:
          agent.hcl: |
            auto_auth {
              method "kubernetes" {
                mount_path = "auth/kubernetes"
                config = {
                  role = "astro-agent"
                }
              }
            }

            template {
              destination = "/vault/secrets/openlineage-api-key"
              contents = "{{ with secret \"secret/data/openlineage/api-key\" }}{{ .Data.data.key }}{{ end }}"
            }
        EOF
        ```

        #### Install the Helm chart with the Vault Agent

        Apply the chart with the `values.yaml` file:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>

    Read more about [secrets backends on Astro](/docs/astro/secrets-backend).
  </Tab>
</Tabs>

### Step 3: Set OpenLineage environment variables on the orchestration plane

<Steps>
  <Step title="Open the Deployment">
    In the Astro UI, open your Deployment.

    #### Open the Environment tab

    Click the **Environment** tab.

    #### Add the OpenLineage variable

    Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet), then click **Add Variable** and add the following environment variable:

    ```sh wrap theme={null}
    OPENLINEAGE_DISABLED=False
    ```

    #### Apply changes

    Click **Update Environment Variables** to save your changes.
  </Step>
</Steps>

Setting this variable ensures that all required OpenLineage events, including task and Dag run events, are collected from the scheduler, workers, Dag processor, and triggerer components. This provides complete lineage in Observe and Astro Alerts.