Skip to main content
Airflow 3 enables users to write SDKs allowing definition of Airflow Dags and tasks in languages other than Python. In Airflow 3.4, experimental SDKs are available for Go, Java, and TypeScript, and Dags can also be written in YAML.
Multilanguage support is currently experimental and under development. This guide is subject to change and will be expanded over time. If you want to contribute to support writing Airflow tasks in the language of your choice, contact the Airflow developers in the Airflow Slack or the Airflow Dev list.
You can also write Dags in YAML with open source packages that turn YAML into Python Dags, at different levels of abstraction. DAG Factory stays close to Airflow primitives, with YAML keys for Dags, tasks, and operators. Blueprint lets you define multi-task templates in Python and use them in YAML.

Assumed knowledge

To get the most out of this guide, you should have existing knowledge of:

How it works

You can use a language SDK in two ways:
  • Full Dag definition: The whole Dag, including its schedule, tasks, and dependencies, is written in Go, Java, or TypeScript. This option was added in Airflow 3.4, and the Go, Java, and TypeScript examples in this guide use it.
  • Python Dag with stub tasks: A Python Dag declares a task with @task.stub(queue="<your-queue>"), and a task handler written in Go, Java, or TypeScript implements it. The dag_id and task_id registered by the task handler need to match the Python Dag. See Python Dags with stub tasks for an example.
For a full Dag definition, you write the Dag with the language SDK and build it into a single file: a Go executable, a Java jar, or a TypeScript .min.mjs file. The SDK is compiled into this file. You place the file in your Dag bundle, for example the dags folder, and the Dag processor starts it with the language runtime. The running file then sends the Dags it contains to the Dag processor. Airflow then schedules the Dag and shows it in the UI like a Python Dag. The Code view shows the source code that was embedded in the file during the build. Each task runs on a coordinator. The coordinator starts the language runtime for the task and passes messages between the task and Airflow, for example to read and write XComs. Two environment variables configure which coordinators are available, in addition to the default Python Task SDK:
  • AIRFLOW__SDK__COORDINATORS: a JSON containing all available coordinators.
  • AIRFLOW__SDK__QUEUE_TO_COORDINATOR: a JSON that maps queue names to coordinator names.
Every task of a Dag written in another language needs a queue that is mapped to the coordinator of that language. The runtime and the coordinator configuration must be available on the Dag processor, which runs the built file to read the Dag, and on the Airflow component that runs your tasks.
airflow dags reserialize (astro dev run dags reserialize in the Astro CLI) and airflow dags test don’t work for Dags written in Go, Java, or TypeScript. Both commands log A native Lang-SDK Dag is parsed only by the Dag processor for each built file. New, renamed, or moved files of Dags written in languages other than Python appear in Airflow after the Dag processor’s next scan of the Dag bundle, which happens every 300 seconds by default ([dag_processor] refresh_interval).
When using task SDKs for other languages on Astro, you need to create matching worker queues in addition to setting AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as environment variables. For example, if you are using the Go SDK, and set AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}', you need to create a worker queue with the name golang.Also make sure any compiled binaries (for example, Go executables) are compatible with linux/amd64, the architecture of workers on Astro.

The example Dag

Each language section in this guide defines the same Dag. In Python, it looks like this:
The extract task returns three orders, transform adds up their amounts, and load prints Total order amount: 100. The values are passed between the tasks as XComs.

Go SDK example

The Go SDK is experimental and still under development. You can track its status here.
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. You need Go 1.27.3 or later to build the Dag.

Step 1: Configure the executable coordinator

Add the following two environment variables. When you run Airflow locally, add them to your .env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.

Step 2: Write the Dag in Go

  1. Create the directory for the Go module:
  2. Create a go.mod file. The go and toolchain lines set the minimum Go version, the require line pins the SDK version, and the tool directive makes the bundle packer available through go tool. Replace <your-go-sdk-version> with the Go SDK version for your Airflow version. You can find the available versions on pkg.go.dev.
  3. Create a main.go file with the Dag:
    airflow.Dag creates the Dag with the id go_etl. The Queue in the DagSpec applies to all tasks of the Dag and routes them to the Go coordinator. dag.Task adds a function as a task, and the task id is the name of the function. airflow.Inputs passes the return value of an upstream task to the function as an argument, which also sets the dependency between the two tasks. The transform function receives the XCom of extract as an orders struct, and its return value becomes its own XCom. The load function writes the result to the task logs with the task logger.
To implement stub tasks of a Python Dag in Go instead, see Python Dags with stub tasks.

Step 3: Build the Dag

From the include/go_etl directory, compile the Dag for the architecture of your Airflow containers (--goos linux for Linux containers) and write it to your dags folder. Use arm64 on Apple Silicon or amd64 on Intel and AMD machines.
This writes a single executable called go_etl to the dags folder. The Dag processor recognizes the file because it has no file extension and ends with the bundle format marker that airflow-go-pack adds.
Astro workers use linux/amd64, which means you’ll need to compile with --goos linux --goarch amd64 before deploying your project to Astro.

Step 4: Run the Dag

After the Dag processor has read the file, the go_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.
When using the Go SDK on Astro, you need to create a matching worker queue in addition to setting AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as environment variables. For example, for AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}', you need to create a worker queue with the name golang.

Java SDK example

The Java SDK is experimental and still under development. You can track its status in the java-sdk directory of the Airflow repository.
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. The Java Dag runs as a compiled jar, so the Dag processor and the Airflow component that runs your tasks also need a Java runtime, for example openjdk-21-jre-headless. Add the runtime to your packages.txt file to install it in your image:
Building the Dag needs a JDK and Gradle.

Step 1: Configure the Java coordinator

Add the following two environment variables. When you run Airflow locally, add them to your .env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.
The coordinator runs java from the PATH by default. To use a different Java installation, set "kwargs": {"java_executable": "<path-to-java>"} in the coordinator entry.

Step 2: Write the Dag in Java

  1. Create the module directory, including the com/example/etl package path for the source file. You run Gradle from the module root, so change into include/java_etl:
  2. Create the Gradle project files. gradle.properties sets the SDK version in one place, settings.gradle names the project, and build.gradle applies the Airflow SDK plugin, pulls in the SDK and its annotation processor, and points airflowBundle at the class with the main method. Replace <your-java-sdk-version> with the Java SDK version for your Airflow version. You can find the available versions on Maven Central. gradle.properties:
    settings.gradle:
    build.gradle:
  3. Create the Dag at src/main/java/com/example/etl/JavaEtl.java:
    The @Airflow.Dag annotation declares the Dag with the id java_etl, and each @Airflow.Task annotation declares a task with its task id and the queue that routes the task to the Java coordinator. The @Airflow.Deps class sets the dependencies. Calling load(transform(extract())) in depends() passes the return value of each task to the next one as an XCom, like the same call in the Python Dag. JavaEtlDeps is generated at compile time by the annotation processor from the annotations on JavaEtl. System.out.println writes to the task logs.
To implement stub tasks of a Python Dag in Java instead, see Python Dags with stub tasks.

Step 3: Build the Dag

From the include/java_etl directory, build the bundle jar with Gradle, then copy it to your dags folder:

Step 4: Run the Dag

Restart your Airflow environment to install the Java runtime. After the Dag processor has read the jar, the java_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.
When using the Java SDK on Astro, you need to create a matching worker queue in addition to setting AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as environment variables. For example, for AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}', you need to create a worker queue with the name java.

TypeScript SDK example

The TypeScript SDK is experimental and still under development. You can track its status in the ts-sdk directory of the Airflow repository.
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. The TypeScript Dag runs on Node.js, so the Dag processor and the Airflow component that runs your tasks need Node.js 22 or later. To install Node.js, copy the node executable from the official Node.js image by adding this line to your Dockerfile:
Building the Dag needs Node.js 22+ and npm.

Step 1: Configure the Node.js coordinator

Add the following two environment variables. When you run Airflow locally, add them to your .env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.

Step 2: Write the Dag in TypeScript

  1. Create the project directory and install the TypeScript SDK and the build tools. Replace <your-ts-sdk-version> with the TypeScript SDK version for your Airflow version. You can find the available versions on npm.
  2. Create the Dag at src/typescript_etl.ts:
    new Dag creates the Dag with the id typescript_etl, and the queue applies to all of its tasks. dag.task adds a function as a task, and the task id is the name of the function. A task function takes one object of keyword arguments, so transform is called with { data: extract() }. Like in the Python Dag, passing the result of one task call to another sets the dependency and passes the value as an XCom. console.log writes to the task logs.
To implement stub tasks of a Python Dag in TypeScript instead, see Python Dags with stub tasks.

Step 3: Build the Dag

From the include/typescript_etl directory, build the Dag into a single file in your dags folder:
The Dag processor recognizes .min.mjs files that airflow-ts-pack created.

Step 4: Run the Dag

Restart your Airflow environment to install Node.js. After the Dag processor has read the file, the typescript_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.

Python Dags with stub tasks

You can also write the Dag in Python and implement only some of its tasks in Go, Java, or TypeScript. The Python Dag declares these tasks as stub tasks, and a task handler in Go, Java, or TypeScript implements each of them. You can mix stub tasks and regular Python tasks in the same Dag, and stub tasks of different languages too. The following example implements each task of the example Dag in a different language: extract in Go, transform in Java, and load in TypeScript.

Step 1: Configure all three coordinators

Configure the coordinators of all three languages with the following two environment variables. When you run Airflow locally, add them to your .env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager. The Java and TypeScript runtimes from the earlier sections also need to be installed.

Step 2: Write the Python Dag

Create a file called polyglot_etl.py in your dags folder:
A stub task has no body. The queue of each stub task routes it to the coordinator of its language, and the task id is the name of the function. The parameters of the stub function define the arguments that the task handler receives. Like in the Python version of the Dag, load(transform(extract())) sets the dependencies and passes the return value of each task to the next one as an XCom.

Step 3: Write the task handlers

A task handler contains the dag_id and task_id of a stub task and the task function. Each language needs its own project for its task handler.
  1. Create a Go project in include/polyglot_go with the go.mod file from Step 2 of the Go example. Then create a main.go file with the task handler of the extract task:
  2. Create a Java project in include/polyglot_java with the Gradle files from Step 2 of the Java example, and set mainClass = "com.example.etl.StubHandlers" in build.gradle. Then create the task handler of the transform task at src/main/java/com/example/etl/StubHandlers.java:
  3. Create a TypeScript project in include/polyglot_typescript with the commands from Step 2 of the TypeScript example. Then create the task handler of the load task at src/polyglot_load.ts:
    The TypeScript task handler receives its arguments as one object, with the names of the parameters of the stub function.

Step 4: Build and run the Dag

  1. Build each project with the build commands from Step 3 of the Go, Java, and TypeScript examples. Give the built files new names in your dags folder, for example polyglot_extract, polyglot_transform.jar, and polyglot_load.min.mjs.
  2. Trigger the polyglot_etl Dag. The logs of the load task show Total order amount: 100, written by the TypeScript task handler.
On Astro, create a worker queue for each queue in AIRFLOW__SDK__QUEUE_TO_COORDINATOR. For this example, you need the worker queues golang, java, and typescript.

YAML Dags

In Airflow 3.4, you can also write Dags as YAML files. The tasks of a YAML Dag either use an existing operator, or run code from a task handler written with a language SDK. YAML Dags are experimental and still under development. See YAML Dags in the Airflow documentation.

Other ways to run tasks in other languages

You can also run tasks in other languages using the following methods: