> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure XCom backend

Remote Execution Agents require a custom XCom backend to pass data between tasks. This guide covers configuring AWS S3, Azure Blob Storage, or GCP Cloud Storage as your XCom backend.

## Overview

XCom (cross-communication) enables data exchange between Airflow tasks. By default, Airflow stores XCom values in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane while agents execute tasks in your infrastructure.

A custom XCom backend stores XCom data in object storage accessible to all agents in your execution plane. This allows tasks running on different agents to share data through a common storage location.

<Warning>
  **XCom limitations with callbacks**

  In Airflow 3, XCom values are stored externally in storage backends. The database only holds the location of each XCom value, and retrieving the actual XCom data requires a call to the execution API with a valid task token.

  Currently, callbacks do not receive a task token, so they cannot retrieve XCom values from external backends. As a result, using XCom in callbacks is not supported at this time when using Remote Execution with custom XCom backends.
</Warning>

<Info>
  **XCom support in triggers**

  XCom triggerer support is enabled with Agent Client 1.5.0+. Triggers running on the agent can retrieve and push XCom values when using a compatible client version.
</Info>

## Prerequisites

* Remote Execution Agent installed and registered
* Object storage bucket in your cloud provider
* Workload identity configured for your Kubernetes cluster (IRSA for AWS, Managed Identity for Azure, Workload Identity for GCP)
* Required Python packages installed in your agent image (see cloud-specific sections below)

## Configure XCom backend

All XCom backend configurations use the `XComObjectStorageBackend` class from the Common IO provider. Choose your cloud provider below for specific configuration instructions.

<Tabs>
  <Tab title="AWS S3" language="aws">
    ### Prerequisites

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-amazon`
    * `apache-airflow-providers-common-io`
    * `s3fs`

    ### Create S3 bucket

    Create an S3 bucket for XCom storage with the following recommended settings:

    * **Versioning**: Enable for data recovery
    * **Encryption**: Enable server-side encryption (SSE-S3 or SSE-KMS)
    * **Lifecycle policy**: Configure automatic deletion of old XCom objects

    ### Configure IAM role

    Create an IAM role with permissions to read and write to your XCom bucket. Attach this policy to the role:

    ```json wrap theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::your-xcom-bucket",
            "arn:aws:s3:::your-xcom-bucket/*"
          ]
        }
      ]
    }
    ```

    Configure the trust relationship to allow your Kubernetes service accounts to assume the role. See [AWS IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) for details.

    ### Update Helm values

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "s3://<connection-id>@<bucket-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>
    ```

    Replace:

    * `<connection-id>`: Airflow connection ID for S3 (for example, `aws_xcom`)
    * `<bucket-name>`: Your S3 bucket name
    * `<path-to-xcom>`: Path prefix for XCom objects (for example, `xcom`)
    * `<AWS_ACCOUNT_ID>`: Your AWS account ID
    * `<IAM_ROLE_NAME>`: IAM role name created above

    ### Configure the AWS connection

    The `<connection-id>` in the XCom path must correspond to a configured Airflow connection. You can define this connection directly in `values.yaml` or through a secrets backend.

    <Tabs>
      <Tab title="values.yaml">
        Add an `AIRFLOW_CONN_<CONNECTION_ID>` environment variable to `commonEnv` in your `values.yaml`. The suffix after `AIRFLOW_CONN_` becomes the connection ID. For example, `AIRFLOW_CONN_AWS_XCOM` creates a connection with ID `aws_xcom`:

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_AWS_XCOM
            value: '{"conn_type":"aws"}'
        ```

        Because the agent uses IRSA for authentication, you don't need to specify explicit credentials in the connection. The IAM role annotation on the pod provides access to S3 automatically.
      </Tab>

      <Tab title="Secrets backend">
        For production environments, store the connection in a [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) instead of `values.yaml`. See [Set up AWS Secrets Manager as your secrets backend](/docs/astro/secrets-backend/aws-secretsmanager#remote-execution) or [Set up AWS Systems Manager Parameter Store as your secrets backend](/docs/astro/secrets-backend/aws-paramstore#remote-execution) for configuration steps.

        When using a secrets backend, create a secret that matches your connection ID. For example, if your XCom path uses `aws_xcom` as the connection ID, store the connection under the path corresponding to `aws_xcom` in your secrets backend.
      </Tab>
    </Tabs>

    ### Apply configuration

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>

  <Tab title="Azure Blob Storage" language="azure">
    ### Prerequisites

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-microsoft-azure`
    * `apache-airflow-providers-common-io`

    ### Create storage account and container

    Create an Azure Storage account and container for XCom storage:

    1. Create a storage account with these recommended settings:
       * **Performance**: Standard
       * **Replication**: Locally-redundant storage (LRS) or higher
       * **Secure transfer**: Required
    2. Create a container named `xcom` (or your preferred name)
    3. Note the storage account name

    ### Configure managed identity

    Create a managed identity and grant it access to your storage account:

    1. Create a managed identity in Azure
    2. Assign the **Storage Blob Data Contributor** role to the managed identity on your storage account
    3. Configure workload identity federation to allow your Kubernetes service accounts to use the managed identity

    See [Azure Workload Identity documentation](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) for details.

    ### Update Helm values

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW_CONN_WASB_XCOM
        value: '{"conn_type":"azure","login":"<storage-account-name>","extra":"{\"anon\": false, \"account_name\":\"<storage-account-name>\",\"managed_identity_client_id\":\"<managed-identity-client-id>\",\"workload_identity_tenant_id\":\"<tenant-id>\"}"}'
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "abfs://wasb_xcom@<container-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    labels:
      azure.workload.identity/use: "true"

    annotations:
      azure.workload.identity/client-id: "<managed-identity-client-id>"
    ```

    Replace:

    * `<storage-account-name>`: Your Azure storage account name
    * `<managed-identity-client-id>`: Client ID of your managed identity
    * `<tenant-id>`: Your Azure tenant ID
    * `<container-name>`: Container name (e.g., `xcom`)
    * `<path-to-xcom>`: Path prefix for XCom objects

    ### Apply configuration

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>

  <Tab title="GCP Cloud Storage" language="gcp">
    ### Prerequisites

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-google`
    * `apache-airflow-providers-common-io`
    * `gcsfs`

    ### Create GCS bucket

    Create a GCS bucket for XCom storage with the following recommended settings:

    * **Location type**: Region or Multi-region based on your requirements
    * **Storage class**: Standard
    * **Access control**: Uniform (bucket-level permissions)
    * **Encryption**: Google-managed or customer-managed encryption keys

    ### Configure IAM service account

    Create a GCP service account and grant it access to your GCS bucket:

    1. Create a service account in your GCP project
    2. Grant the service account the **Storage Object Admin** role (`roles/storage.objectAdmin`) on your GCS bucket
    3. Configure Workload Identity to allow your Kubernetes service accounts to use the GCP service account

    To link your Kubernetes service account (KSA) to your GCP service account (GSA):

    ```sh wrap theme={null}
    gcloud iam service-accounts add-iam-policy-binding <your-service-account>@<project>.iam.gserviceaccount.com \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:<project>.svc.id.goog[<namespace>/<ksa-name>]"
    ```

    See [GKE Workload Identity documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for details.

    ### Update Helm values

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "gs://<bucket-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    annotations:
      iam.gke.io/gcp-service-account: <your-service-account>@<project>.iam.gserviceaccount.com
    ```

    Replace:

    * `<bucket-name>`: Your GCS bucket name
    * `<path-to-xcom>`: Path prefix for XCom objects (e.g., `xcom`)
    * `<your-service-account>`: Your GCP service account name
    * `<project>`: Your GCP project ID

    ### Apply configuration

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>

## Configuration options

### XCom path format

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` parameter defines where XCom objects are stored:

```text wrap theme={null}
<protocol>://<connection-id>@<bucket-or-container>/<path-prefix>
```

* **protocol**: `s3://` for AWS, `abfs://` for Azure, `gs://` for GCP
* **connection-id**: Airflow connection ID for authentication
* **bucket-or-container**: Storage bucket or container name
* **path-prefix**: Optional path prefix for organizing XCom objects

### XCom threshold

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD` parameter controls when XCom values are stored in object storage:

* `0`: Always store in object storage (recommended for Remote Execution)
* `>0`: Store in object storage only if value size exceeds threshold in bytes

Set this to `0` to ensure all XCom values are accessible to remote agents.

### Compression

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION` parameter controls compression:

* `zip`: Compress XCom values (recommended)
* `None`: No compression

Compression reduces storage costs and transfer time for large XCom values.

## Verify configuration

After applying your configuration, verify XCom backend functionality:

1. Trigger a test DAG that uses XCom to pass data between tasks
2. Check that tasks complete successfully
3. Verify XCom objects appear in your storage bucket
4. Review agent logs for any XCom-related errors

## Next steps

After configuring XCom backend:

* [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend) - Store Airflow connections and variables securely
* [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend) - Store task and asset state on Astro Runtime 3.3 and later
* [Configure dag sources](/docs/astro/remote-execution-configure-dag-sources) - Define how agents access DAG code
* [Configure logging](/docs/astro/remote-execution-logging-overview) - Export task logs to external platforms

## Related documentation

* [XCom backend strategies](/docs/learn/custom-xcom-backend-strategies)
* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)