Configure XCom backend for a Remote Execution Agent

Airflow 3

This feature is only available for Airflow 3.x Deployments.

XCom (cross-communication) enables you to move data between tasks, or cross communicate between tasks in Airflow. By default, Airflow uses the metadata database to store XComs and you can configure a custom XCom backend for storing XComs elsewhere. When using Remote Execution, configuring a custom XCom backend is required because the Airflow metadata database exists in the Astro Orchestration Plane, not in the Execution Plane where the Remote Execution Agents live.

This procedure describes how to set up the XCom Backend in the Remote Execution Agent components.

Basic configuration

In your Agent’s values.yaml file, set the following configuration. Note that your Agent’s Docker image requires apache-airflow-providers-common-io to be installed.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

The recommendations described in the Strategies for custom XCom backends in Airflow also apply to Remote Execution Agents.

However, instead of configuring environment variables such as AIRFLOW_CORE_XCOM_BACKEND in the Astro UI, you must configure settings in your Helm values.yaml to apply these settings to your Remote Execution agents.

Example: Amazon S3 custom XCom backend configuration

The Remote Execution Agent image must include Python packages apache-airflow-providers-amazon and s3fs.

The following example shows the configuration you can use if you use Amazon S3 in your environment.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2 
3 commonEnv:
4   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
5     value: "s3://<connection-id>@<bucket>/<path-to-xcom>"
6   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
7     value: "0"  # Zero means always store in the object store
8   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
9     value: "zip"
10 
11 annotations:
12   eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/AstronomerAgentRole

For details on Airflow S3 connection fields, see the Airflow Amazon S3 connection documentation. For guidance on configuring IAM Roles for Service Accounts (IRSA) in EKS, see the AWS EKS documentation.

Example: Azure Blob Storage custom XCom backend configuration

The following example shows the configuration you can use if you use Azure Blob Storage with managed identities in your environment.

The Remote Execution Agent image must include the Microsoft Azure provider for Airflow.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2 
3 commonEnv:
4   - name: AIRFLOW_CONN_WASB_XCOM # Defines the Azure Blob Storage connection
5     value: '{"conn_type":"azure","login":"<storage_account_name>","extra":"{\"anon\": false, \"account_name\":\"<storage_account_name>\",\"managed_identity_client_id\":\"<your-managed-identity-client-id>\",\"workload_identity_tenant_id\":\"<your-tenant-id>\"}"}'
6   - name: AIRFLOW__CORE__LOAD_EXAMPLES
7     value: "True"
8   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
9     value: abfs://wasb_xcom@<container>/<path>
10   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
11     value: "0" # Zero means always store in the object store
12   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
13     value: "zip"
14 
15 labels:
16   azure.workload.identity/use: "true"
17 
18 annotations:
19   azure.workload.identity/client-id: "<managed-identity-client-id>"

For details on Airflow Azure connection fields, see the Airflow Azure integration docs. For guidance on managed identities in Azure, see the Microsoft documentation.