Configure XCOM backend for a Remote Execution Agent

Airflow 3

This feature is only available for Airflow 3.x Deployments.

XCom (cross-communication) enables you to move data between tasks, or cross communicate between tasks in Airflow. By default, Airflow uses the metadata database to store XComs, but you can configure a custom XCom backend. Because the Airflow metadata database exists in the Astro Orchestration Plane, not in the Execution Plane with the Remote Execution Agent, you must configure a custom XCom Backend to make optimal use of them in your environment.

This procedure describes how to set up the XCom Backend in the Remote Execution Agent components.

Basic configuration

In your Agent’s values.yaml file, set the following configuration.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

You can use the recommendations described in the Strategies for custom XCom backends in Airflow with the Remote Execution Agent.

However, there is one difference. Instead of directly setting the AIRFLOW_CORE_XCOM_BACKEND environment variable, the xcomBackend field in your values.yaml file configures this setting for you.:

1 xcomBackend: "AIRFLOW__CORE__XCOM_BACKEND"

Example: Amazon S3 custom XCom backend configuration

The Remote Execution Agent image must include the Amazon Web Services (AWS) provider for Airflow.

The following example shows the configuration you can use if you use Amazon S3 in your environment.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2 
3 commonEnv:
4   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
5     value: "s3://<connection-id>@<bucket>/<path-to-xcom>"
6   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
7     value: "0"  # Zero means always store in the object store
8   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
9     value: "zip"
10 
11 annotations:
12   eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/AstronomerAgentRole

For details on Airflow S3 connection fields, see the Airflow Amazon S3 connection documentation. For guidance on configuring IAM Roles for Service Accounts (IRSA) in EKS, see the AWS EKS documentation.

Example: Azure Blob Storage custom XCom backend configuration

The following example shows the configuration you can use if you use Azure Blob Storage with managed identities in your environment.

The Remote Execution Agent image must include the Microsoft Azure provider for Airflow.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2 
3 commonEnv:
4   - name: AIRFLOW_CONN_WASB_XCOM # Defines the Azure Blob Storage connection
5     value: '{"conn_type":"azure","login":"<storage_account_name>","extra":"{\"anon\": false, \"account_name\":\"<storage_account_name>\",\"managed_identity_client_id\":\"<your-managed-identity-client-id>\",\"workload_identity_tenant_id\":\"<your-tenant-id>\"}"}'
6   - name: AIRFLOW__CORE__LOAD_EXAMPLES
7     value: "True"
8   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
9     value: abfs://wasb_xcom@<container>/<path>
10   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
11     value: "0" # Zero means always store in the object store
12   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
13     value: "zip"
14 
15 labels:
16   azure.workload.identity/use: "true"
17 
18 annotations:
19   azure.workload.identity/client-id: "<managed-identity-client-id>"

For details on Airflow Azure connection fields, see the Airflow Azure integration docs. For guidance on managed identities in Azure, see the Microsoft documentation.