> ## Documentation Index > Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Set up a custom XCom backend using object storage By default, Airflow uses the [metadata database](/docs/learn/2.x/airflow-database) to store XComs, which works well for local development but has limited performance. For production environments that use XCom to pass data between tasks, Astronomer recommends using a custom XCom backend. [Custom XCom backends](/docs/learn/2.x/custom-xcom-backend-strategies) allow you to configure where Airflow stores information that is passed between tasks using [XComs](/docs/learn/2.x/airflow-passing-data-between-tasks#xcom). The Object Storage XCom Backend available in the [Common IO provider](https://airflow.apache.org/docs/apache-airflow-providers-common-io/stable/index.html) is the easiest way to store XComs in a remote object storage solution. This tutorial will show you how to set up a custom XCom backend using object storage for [AWS S3](https://aws.amazon.com/s3/), [GCP Cloud Storage](https://cloud.google.com/storage) or [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/). To learn more about other options for setting custom XCom backends, see [Strategies for custom XCom backends in Airflow](/docs/learn/2.x/custom-xcom-backend-strategies). While a custom XCom backend allows you to store virtually unlimited amounts of data as XComs, you will also need to scale other Airflow components to pass large amounts of data between tasks. For help running Airflow at scale, [contact Astronomer](https://www.astronomer.io/lp/signup/?referral=docs-content-link\&utm_medium=docs\&utm_content=learn-xcom-backend-tutorial\&utm_source=body). Object storage is currently considered experimental and might be subject to breaking changes in future releases. For more information see [AIP-58](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565). ## Time to complete This tutorial takes approximately 45 minutes to complete. ## Assumed knowledge To get the most out of this tutorial, make sure you have an understanding of: * XCom basics. See [Passing data between Airflow tasks](/docs/learn/2.x/airflow-passing-data-between-tasks). * Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/2.x/connections). ## Prerequisites * The [Astro CLI](https://docs.astronomer.io/astro/cli/install-cli) with an Astro project running Astro Runtime 11.5.0 or higher (Airflow 2.9.2 or higher). To set up a custom XCom backend with older versions of Airflow, see [Custom XCom backends](/docs/learn/2.x/custom-xcom-backend-strategies). * An account in either [AWS](https://aws.amazon.com/), [GCP](https://cloud.google.com/), or [Azure](https://azure.microsoft.com/) with permissions to create and configure an object storage container. ## Step 1: Set up your object storage container First, you need to set up the object storage container in your cloud provider where Airflow will store the XComs.
AWS 1. Log into your AWS account and [create a new S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). Ensure that public access to the bucket is blocked. You don't need to enable bucket versioning. 2. [Create a new IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) for Airflow to access your bucket. You can use the JSON configuration below or use the AWS GUI to replicate what you see in the screenshot. Replace `` with the name of your S3 bucket. ```json wrap theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:ReplicateObject", "s3:PutObject", "s3:GetObject", "s3:RestoreObject", "s3:ListBucket", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::/*", "arn:aws:s3:::" ] } ] } ``` 3. Save your policy under the name `AirflowXComBackendAWSS3`. 4. [Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) called `airflow-xcom` with the AWS credential type `Access key - Programmatic access` and [attach](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) the `AirflowXComBackendAWSS3` policy to this user. 5. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) of the type `Third-party-service` for your `airflow-xcom` user. Make sure to save the Access Key ID and the Secret Access Key in a secure location to use in [Step 3](#step-3-set-up-your-airflow-connection). For other ways to set up a connection between Airflow and AWS, see the [Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) documentation.
GCP 1. Log into your Google Cloud account and [create a new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects). 2. [Create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) in your project with Uniform Access Control. Enforce public access prevention. 3. [Create a custom IAM role](https://cloud.google.com/iam/docs/creating-custom-roles) called `AirflowXComBackendGCS` for Airflow to access your bucket. Assign 6 permissions: * `storage.buckets.list` * `storage.objects.create` * `storage.objects.delete` * `storage.objects.get` * `storage.objects.list` * `storage.objects.update` 4. [Create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) called `airflow-xcom` and grant it access to your project by granting it the `AirflowXComBackendGCS` role. 5. [Create a new key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your `airflow-xcom` service account and make sure to download the credentials in JSON format. For other ways to set up a connection between Airflow and Google Cloud, see the [Google provider](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html) documentation.
Azure 1. Log into your Azure account and [create a storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create). Ensure that public access to the bucket is blocked. 2. In the storage account, [create a new container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal). 3. [Create a shared access token](https://learn.microsoft.com/en-us/azure/cognitive-services/Translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers) for your container. In the **Permissions** dropdown menu, enable the following permissions: * Read * Add * Create * Write * Delete * List Set the duration the token will be valid and set **Allowed Protocols** to `HTTPS only`. Provide the IP address of your Airflow instance. If you are running Airflow locally with the Astro CLI, use the IP address of your computer. 4. Go to your Storage account and navigate to [**Access keys**](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage). Copy the **Key** and **Connection string** values and save them in a secure location to use in [step 3](#step-3-set-up-your-airflow-connection). For other ways to set up a connection between Airflow and Azure Blob Storage, see the [Microsoft Azure provider](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html) documentation.
## Step 2: Install the required provider packages To use the Object Storage XCom Backend, you need to install the Common IO provider package and the provider package for your object storage container provider.
AWS Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Amazon](https://airflow.apache.org/registry/providers/amazon/) provider packages to your `requirements.txt` file. Note that you need to install the `s3fs` extra to use the Amazon provider package with the [object storage](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html) feature. ```text wrap theme={null} apache-airflow-providers-common-io==1.3.2 apache-airflow-providers-amazon[s3fs]==8.19.0 ```
GCP Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Google](https://airflow.apache.org/registry/providers/google/) provider packages to your `requirements.txt` file. ```text wrap theme={null} apache-airflow-providers-common-io==1.3.2 apache-airflow-providers-google==10.17.0 ```
Azure Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Microsoft Azure](https://airflow.apache.org/registry/providers/microsoft-azure/) provider packages to your `requirements.txt` file. ```text wrap theme={null} apache-airflow-providers-common-io==1.3.2 apache-airflow-providers-microsoft-azure==9.0.0 ```
## Step 3: Set up your Airflow connection An Airflow connection is necessary to connect Airflow with your object storage container provider. In this tutorial, you'll use the Airflow UI to configure your connection. 1. Start your Astro project by running: ```bash wrap theme={null} astro dev start ```
AWS 2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields: * **Conn Id**: `my_aws_conn` * **Conn Type**: `Amazon Web Services` * **AWS Access Key ID**: `` * **AWS Secret Access Key**: `` To learn more about configuration options for the AWS connection, see the [Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) documentation.
GCP 2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields: * **Conn Id**: `my_gcp_conn` * **Conn Type**: `Google Cloud` * **Project Id**: `` * **Keyfile JSON**: `` To learn more about configuration options for the Google connection, see the [Google provider](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html) documentation.
Azure 2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields: * **Conn Id**: `my_azure_conn` * **Conn Type**: `Microsoft Azure Blob Storage` * **Account URL (Active Directory Auth)**: `` * **Blob Storage Key (optional)**: `` * **Blob Storage Connection String (optional)**: `` To learn more about configuration options for the Azure connection, see the [Microsoft Azure provider](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html) documentation.
## Step 4: Configure your custom XCom backend Configuring a custom XCom backend with object storage can be done by setting environment variables in your Astro project. If you are setting up a custom XCom backend for an Astro deployment, you have to set the following environment variables for your deployment. See [Environment variables](https://docs.astronomer.io/astro/environment-variables) for instructions. 1. Add the `AIRFLOW__CORE__XCOM_BACKEND` environment variable to your `.env` file. It defines the class to use for the custom XCom backend implementation. ```text wrap theme={null} AIRFLOW__CORE__XCOM_BACKEND="airflow.providers.common.io.xcom.backend.XComObjectStorageBackend" ```
AWS 2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your S3 bucket where the XComs will be stored in the form of `@/`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `` with your S3 bucket name. ```text wrap theme={null} AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="s3://my_aws_conn@/xcom" ```
GCP 2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your GCS bucket where the XComs will be stored in the form of `@/`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `` with your GCS bucket name. ```text wrap theme={null} AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="gs://my_gcp_conn@/xcom" ```
Azure 2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your Azure blob container where the XComs will be stored in the form of `@/`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `` with your Azure blob container name. ```text wrap theme={null} AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="abfs://my_azure_conn@/xcom" ```
3. Add the `AIRFLOW__COMMON.IO__XCOM_OBJECTSTORAGE_THRESHOLD` environment variable to your `.env` file to determine when Airflow will store XComs in the object storage vs the metadata database. The default value is `-1` which will store all XComs in the metadata database. Set the value to `0` to store all XComs in the object storage. Any positive value means any XCom with a byte size greater than the threshold will be stored in the object storage and any XCom with a size equal to or less than the threshold will be stored in the metadata database. For this tutorial we will set the threshold to `1000` bytes, which means any XCom larger than 1KB will be stored in the object storage. ```text wrap theme={null} AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD="1000" ``` 4. Optional. Define the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORE_COMPRESSION` environment variable to compress the XComs stored in the object storage with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) supported compression algorithms like `zip`. The default value is `None`. ```text wrap theme={null} AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION="zip" ``` 5. Restart your Airflow project by running: ```bash wrap theme={null} astro dev restart ``` ## Step 5: Test your custom XCom backend We will use a simple DAG to test your custom XCom backend. 1. Create a new file in the `dags` directory of your Astro project called `custom_xcom_backend_test.py` and add the following code: ```python expandable wrap theme={null} """ ## Toy DAG to show size dependant custom XCom serialization This DAG pushes two dicts to XCom, one below, one above 1000 bytes. It then pulls them and prints their sizes. """ from airflow.decorators import dag, task from airflow.models.baseoperator import chain @dag( start_date=None, schedule=None, catchup=False, doc_md=__doc__, tags=["xcom", "2-9", "toy"], ) def custom_xcom_backend_test(): @task def push_objects(**context) -> None: """Create a small and a big dictionary, print their sizes and push them to XCom.""" small_obj = {"a": 23} big_obj = {f"key{i}": "x" * 100 for i in range(100)} print(f"Size of small object: {small_obj.__sizeof__()}") print(f"Size of big object: {big_obj.__sizeof__()}") context["ti"].xcom_push(key="small_obj", value=small_obj) context["ti"].xcom_push(key="big_obj", value=big_obj) @task def pull_objects(**context) -> None: """Pull the small and big dictionaries from XCom and print their sizes.""" small_obj = context["ti"].xcom_pull(task_ids="push_objects", key="small_obj") big_obj = context["ti"].xcom_pull(task_ids="push_objects", key="big_obj") print(f"Size of small object: {small_obj.__sizeof__()}") print(f"Size of big object: {big_obj.__sizeof__()}") chain(push_objects(), pull_objects()) custom_xcom_backend_test() ``` 2. Manually trigger the `custom_xcom_backend_test` DAG in the Airflow UI and navigate to the XCom tab of the `push_objects` task. You should see that the `small_obj` XCom shows its value, meaning it was stored in the metadata database, since it is smaller than 1KB. The `big_obj` XCom shows the path to the object in the object storage containing the serialized value of the XCom. XCom tab of the push_objects task showing two key-value pairs showing the "big_obj" being serialized to the custom XCom backend and the "small_obj": a dictionary containing 'a': 23, which was stored in the metadata database. ## Conclusion Congratulations, you learned how to set up a custom XCom backend using object storage! Learn more about other options to set up custom XCom backends in the [Strategies for custom XCom backends in Airflow](/docs/learn/2.x/custom-xcom-backend-strategies) guide.