Configure XCOM backend for a Remote Execution Agent

Airflow 3

This feature is only available for Airflow 3.x Deployments.

XCom (cross-communication) enables you to move data between tasks, or cross communicate between tasks in Airflow. By default, Airflow uses the metadata database to store XComs, but you can configure a custom XCom backend. Because the Airflow metadata database exists in the Astro Orchestration Plane, not in the Execution Plane with the Remote Execution Agent, you must configure a custom XCom Backend to make optimal use of them in your environment.

This procedure describes how to set up the XCom Backend in the Remote Execution Agent components.

Basic configuration

In your Agent’s values.yaml file, set the following configuration.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

You can use the recommendations described in the Strategies for custom XCom backends in Airflow with the Remote Execution Agent.

However, there is one difference. Instead of directly setting the AIRFLOW_CORE_XCOM_BACKEND environment variable, the xcomBackend field in your values.yaml file configures this setting for you.:

1 xcomBackend: "AIRFLOW__CORE__XCOM_BACKEND"

Example: Amazon S3 custom XCom backend configuration

The following example shows the configuration you can use if you use Amazon S3 in your environment.

1 xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2 
3 commonEnv:
4   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
5     value: "s3://<connection-id>@<bucket>/<path-to-xcom>"
6   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
7     value: "0"  # Zero means always store in the object store
8   - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
9     value: "zip"