Initialize and deploy Remote Execution projects

This is feature is only available if you are on the Enterprise tier or above. See Astro Plans and Pricing.

Airflow 3

This feature is only available for Airflow 3.x Deployments.

Remote Execution Deployments require two sets of container images:

Astro Runtime images: Astronomer’s managed distribution of Apache Airflow, powering all Deployments on Astro.
Remote Execution Agent images: Specifically built for Remote Execution Agents.

When using Remote Execution mode, the server (Deployment on Astro) and client (Remote Execution Agents) are deployed separately. You must maintain dedicated client images for the Remote Execution Agents. These client images are based on the Astro Remote Execution Agent image and must be pushed to a remote image registry accessible by your Agents.

Because these client images are distinct from the usual Astro Runtime images, you need to use dedicated commands to build and deploy them.

This guide explains how to initialize an Astro project for Remote Execution and how to build and push client images for Remote Execution Agents.

For details about Astro project image deploys with Remote Execution, see Deploy an Astro project.

For best practices and procedures to upgrade your Astro Runtime version, including for Remote Execution Deployments, see Upgrade Astro Runtime.

Prerequisites

The Astro CLI.
An Astro Workspace with at least one Remote Execution Deployment.
Remote Execution Agents registered with your Deployment.
A remote Docker registry accessible from your environment where Remote Execution Agents run.

Create a Remote Execution project

Run:

1 astro dev init my-remote-project --remote-execution-enabled --remote-image-repository <remote-image-repository-url>

This generates the required files for building client images. If you omit --remote-image-repository, you can configure it later with:

1 astro config set remote.client_registry <remote-image-repository-url>

Review generated files

Dockerfile.client – Dockerfile for client images
requirements-client.txt – Python dependencies for client images
packages-client.txt – OS-level packages for client images

Configure dag access

Set up dag access for your Remote Execution Agents using either LocalDagBundle or GitDagBundle. Update your Agent’s Helm chart values.yaml accordingly. See Dag bundles.

Log in to your remote Docker registry

Before you can deploy a client image, you must be logged in to your remote Docker registry. The Astro CLI does not manage authentication for your registry.

1 docker login <remote-registry> --username <registry-username> --password-stdin

Replace <remote-registry> and <registry-username> with the details for your registry.

Build and deploy the Astro Deployment (server) image

From your project directory, deploy the Astro Runtime image to your Remote Execution Deployment:

1 astro deploy

This command builds your project and pushes the updated Astro Runtime image to Astro, updating the Airflow components in your managed Astro Deployment.

Monitor the Deployment to confirm that the new version is running in the Astro UI.

The astro deploy command updates the orchestration plane only. To update the execution plane, follow the Remote Execution Agent image steps below.

Build and deploy Remote Execution Agent (client) images

From your project directory, run:

1 astro remote deploy

You can use additional flags as needed, such as:

1 astro remote deploy --platform linux/amd64
2 astro remote deploy --platform linux/amd64,linux/arm64
3 astro remote deploy --image-name my-custom-image:tag
4 astro remote deploy --build-secrets id=mysecret,src=secrets.txt

Platform mismatch can cause build failures

By default, astro remote deploy builds an image for your computer’s platform. For example, if you are on a Mac with Apple silicon, it will build for linux/arm64. If your Agents need linux/amd64, the image may not work.

To avoid issues, always set the --platform flag: astro remote deploy --platform linux/amd64,linux/arm64

This command only pushes the client image to your remote registry. See astro remote for more information.

Update Remote Execution Agents to use the new image

After pushing the image, you must update your Agent configuration to actually use it:

Edit your Agent Helm chart values.yaml.
Update the image field for each of the worker, dagProcessor, and triggerer component sections to the new client image reference (as just pushed).
Apply the changes by running:

1 helm upgrade <release-name> <chart-path> -f values.yaml

Agents will not use the new client image until this update is performed.

If you manage your Agent deployments via a CI/CD pipeline, update your pipeline to set the new client image in the Helm values.yaml file, and trigger the deployment process to roll out the change automatically. This ensures that updates to the client image are reliably applied across all agents without manual intervention.