Manage DeploymentsRemote ExecutionCore configuration

Configure XCom backend

Remote Execution Agents require a custom XCom backend to pass data between tasks. This guide covers configuring AWS S3, Azure Blob Storage, or GCP Cloud Storage as your XCom backend.

Overview

XCom (cross-communication) enables data exchange between Airflow tasks. By default, Airflow stores XCom values in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane while agents execute tasks in your infrastructure.

A custom XCom backend stores XCom data in object storage accessible to all agents in your execution plane. This allows tasks running on different agents to share data through a common storage location.

XCom limitations with triggers and callbacks

In Airflow 3, XCom values are stored externally in storage backends. The database only holds the location of each XCom value, and retrieving the actual XCom data requires a call to the execution API with a valid task token.

Currently, triggers and callbacks do not receive a task token, so they cannot retrieve XCom values from external backends. As a result, using XCom in triggers or callbacks is not supported at this time when using Remote Execution with custom XCom backends.

Prerequisites

  • Remote Execution Agent installed and registered
  • Object storage bucket in your cloud provider
  • Workload identity configured for your Kubernetes cluster (IRSA for AWS, Managed Identity for Azure, Workload Identity for GCP)
  • Required Python packages installed in your agent image (see cloud-specific sections below)

Configure XCom backend

All XCom backend configurations use the XComObjectStorageBackend class from the Common IO provider. Choose your cloud provider below for specific configuration instructions.

Prerequisites

The Remote Execution Agent image must include these Python packages:

  • apache-airflow-providers-amazon
  • apache-airflow-providers-common-io
  • s3fs

Create S3 bucket

Create an S3 bucket for XCom storage with the following recommended settings:

  • Versioning: Enable for data recovery
  • Encryption: Enable server-side encryption (SSE-S3 or SSE-KMS)
  • Lifecycle policy: Configure automatic deletion of old XCom objects

Configure IAM role

Create an IAM role with permissions to read and write to your XCom bucket. Attach this policy to the role:

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Action": [
7        "s3:GetObject",
8        "s3:PutObject",
9        "s3:DeleteObject",
10        "s3:ListBucket"
11      ],
12      "Resource": [
13        "arn:aws:s3:::your-xcom-bucket",
14        "arn:aws:s3:::your-xcom-bucket/*"
15      ]
16    }
17  ]
18}

Configure the trust relationship to allow your Kubernetes service accounts to assume the role. See AWS IRSA documentation for details.

Update Helm values

Update your agent’s values.yaml file with the XCom backend configuration:

1xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2
3commonEnv:
4  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
5    value: "s3://<connection-id>@<bucket-name>/<path-to-xcom>"
6  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
7    value: "0"  # Always store XComs in object storage
8  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
9    value: "zip"
10
11annotations:
12  eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>

Replace:

  • <connection-id>: Airflow connection ID for S3 (use aws_default or create a custom connection)
  • <bucket-name>: Your S3 bucket name
  • <path-to-xcom>: Path prefix for XCom objects (e.g., xcom)
  • <AWS_ACCOUNT_ID>: Your AWS account ID
  • <IAM_ROLE_NAME>: IAM role name created above

Apply configuration

Update your Helm release:

$helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configuration options

XCom path format

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH parameter defines where XCom objects are stored:

<protocol>://<connection-id>@<bucket-or-container>/<path-prefix>
  • protocol: s3:// for AWS, abfs:// for Azure, gs:// for GCP
  • connection-id: Airflow connection ID for authentication
  • bucket-or-container: Storage bucket or container name
  • path-prefix: Optional path prefix for organizing XCom objects

XCom threshold

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD parameter controls when XCom values are stored in object storage:

  • 0: Always store in object storage (recommended for Remote Execution)
  • >0: Store in object storage only if value size exceeds threshold in bytes

Set this to 0 to ensure all XCom values are accessible to remote agents.

Compression

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION parameter controls compression:

  • zip: Compress XCom values (recommended)
  • None: No compression

Compression reduces storage costs and transfer time for large XCom values.

Verify configuration

After applying your configuration, verify XCom backend functionality:

  1. Trigger a test DAG that uses XCom to pass data between tasks
  2. Check that tasks complete successfully
  3. Verify XCom objects appear in your storage bucket
  4. Review agent logs for any XCom-related errors

Next steps

After configuring XCom backend: