Assumed knowledge
To get the most out of this guide, you should have existing knowledge of:- Basic Airflow concepts. See Introduction to Apache Airflow.
- Depending on the language you use:
- Basic Go concepts. See Go documentation.
- Basic Java concepts. See Learn Java.
- Basic TypeScript concepts. See TypeScript documentation.
How it works
You can use a language SDK in two ways:- Full Dag definition: The whole Dag, including its schedule, tasks, and dependencies, is written in Go, Java, or TypeScript. This option was added in Airflow 3.4, and the Go, Java, and TypeScript examples in this guide use it.
- Python Dag with stub tasks: A Python Dag declares a task with
@task.stub(queue="<your-queue>"), and a task handler written in Go, Java, or TypeScript implements it. Thedag_idandtask_idregistered by the task handler need to match the Python Dag. See Python Dags with stub tasks for an example.
.min.mjs file. The SDK is compiled into this file. You place the file in your Dag bundle, for example the dags folder, and the Dag processor starts it with the language runtime. The running file then sends the Dags it contains to the Dag processor. Airflow then schedules the Dag and shows it in the UI like a Python Dag. The Code view shows the source code that was embedded in the file during the build.
Each task runs on a coordinator. The coordinator starts the language runtime for the task and passes messages between the task and Airflow, for example to read and write XComs. Two environment variables configure which coordinators are available, in addition to the default Python Task SDK:
AIRFLOW__SDK__COORDINATORS: a JSON containing all available coordinators.AIRFLOW__SDK__QUEUE_TO_COORDINATOR: a JSON that mapsqueuenames to coordinator names.
queue that is mapped to the coordinator of that language.
The runtime and the coordinator configuration must be available on the Dag processor, which runs the built file to read the Dag, and on the Airflow component that runs your tasks.
airflow dags reserialize (astro dev run dags reserialize in the Astro CLI) and airflow dags test don’t work for Dags written in Go, Java, or TypeScript. Both commands log A native Lang-SDK Dag is parsed only by the Dag processor for each built file. New, renamed, or moved files of Dags written in languages other than Python appear in Airflow after the Dag processor’s next scan of the Dag bundle, which happens every 300 seconds by default ([dag_processor] refresh_interval).The example Dag
Each language section in this guide defines the same Dag. In Python, it looks like this:extract task returns three orders, transform adds up their amounts, and load prints Total order amount: 100. The values are passed between the tasks as XComs.
Go SDK example
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. You need Go 1.27.3 or later to build the Dag.Step 1: Configure the executable coordinator
Add the following two environment variables. When you run Airflow locally, add them to your.env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.
Step 2: Write the Dag in Go
-
Create the directory for the Go module:
-
Create a
go.modfile. Thegoandtoolchainlines set the minimum Go version, therequireline pins the SDK version, and thetooldirective makes the bundle packer available throughgo tool. Replace<your-go-sdk-version>with the Go SDK version for your Airflow version. You can find the available versions on pkg.go.dev. -
Create a
main.gofile with the Dag:airflow.Dagcreates the Dag with the idgo_etl. TheQueuein theDagSpecapplies to all tasks of the Dag and routes them to the Go coordinator.dag.Taskadds a function as a task, and the task id is the name of the function.airflow.Inputspasses the return value of an upstream task to the function as an argument, which also sets the dependency between the two tasks. Thetransformfunction receives the XCom ofextractas anordersstruct, and its return value becomes its own XCom. Theloadfunction writes the result to the task logs with the task logger.
To implement stub tasks of a Python Dag in Go instead, see Python Dags with stub tasks.
Step 3: Build the Dag
From theinclude/go_etl directory, compile the Dag for the architecture of your Airflow containers (--goos linux for Linux containers) and write it to your dags folder. Use arm64 on Apple Silicon or amd64 on Intel and AMD machines.
go_etl to the dags folder. The Dag processor recognizes the file because it has no file extension and ends with the bundle format marker that airflow-go-pack adds.
Astro workers use
linux/amd64, which means you’ll need to compile with --goos linux --goarch amd64 before deploying your project to Astro.Step 4: Run the Dag
After the Dag processor has read the file, thego_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.
When using the Go SDK on Astro, you need to create a matching worker queue in addition to setting
AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as environment variables. For example, for AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}', you need to create a worker queue with the name golang.Java SDK example
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. The Java Dag runs as a compiled jar, so the Dag processor and the Airflow component that runs your tasks also need a Java runtime, for exampleopenjdk-21-jre-headless. Add the runtime to your packages.txt file to install it in your image:
Step 1: Configure the Java coordinator
Add the following two environment variables. When you run Airflow locally, add them to your.env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.
java from the PATH by default. To use a different Java installation, set "kwargs": {"java_executable": "<path-to-java>"} in the coordinator entry.
Step 2: Write the Dag in Java
-
Create the module directory, including the
com/example/etlpackage path for the source file. You run Gradle from the module root, so change intoinclude/java_etl: -
Create the Gradle project files.
gradle.propertiessets the SDK version in one place,settings.gradlenames the project, andbuild.gradleapplies the Airflow SDK plugin, pulls in the SDK and its annotation processor, and pointsairflowBundleat the class with themainmethod. Replace<your-java-sdk-version>with the Java SDK version for your Airflow version. You can find the available versions on Maven Central.gradle.properties:settings.gradle:build.gradle: -
Create the Dag at
src/main/java/com/example/etl/JavaEtl.java:The@Airflow.Dagannotation declares the Dag with the idjava_etl, and each@Airflow.Taskannotation declares a task with its task id and thequeuethat routes the task to the Java coordinator. The@Airflow.Depsclass sets the dependencies. Callingload(transform(extract()))independs()passes the return value of each task to the next one as an XCom, like the same call in the Python Dag.JavaEtlDepsis generated at compile time by the annotation processor from the annotations onJavaEtl.System.out.printlnwrites to the task logs.
To implement stub tasks of a Python Dag in Java instead, see Python Dags with stub tasks.
Step 3: Build the Dag
From theinclude/java_etl directory, build the bundle jar with Gradle, then copy it to your dags folder:
Step 4: Run the Dag
Restart your Airflow environment to install the Java runtime. After the Dag processor has read the jar, thejava_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.
When using the Java SDK on Astro, you need to create a matching worker queue in addition to setting
AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as environment variables. For example, for AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}', you need to create a worker queue with the name java.TypeScript SDK example
Make sure your Airflow project is at least on version 3.4 and using the Task SDK version 1.4+. The TypeScript Dag runs on Node.js, so the Dag processor and the Airflow component that runs your tasks need Node.js 22 or later. To install Node.js, copy thenode executable from the official Node.js image by adding this line to your Dockerfile:
npm.
Step 1: Configure the Node.js coordinator
Add the following two environment variables. When you run Airflow locally, add them to your.env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager.
Step 2: Write the Dag in TypeScript
-
Create the project directory and install the TypeScript SDK and the build tools. Replace
<your-ts-sdk-version>with the TypeScript SDK version for your Airflow version. You can find the available versions on npm. -
Create the Dag at
src/typescript_etl.ts:new Dagcreates the Dag with the idtypescript_etl, and thequeueapplies to all of its tasks.dag.taskadds a function as a task, and the task id is the name of the function. A task function takes one object of keyword arguments, sotransformis called with{ data: extract() }. Like in the Python Dag, passing the result of one task call to another sets the dependency and passes the value as an XCom.console.logwrites to the task logs.
To implement stub tasks of a Python Dag in TypeScript instead, see Python Dags with stub tasks.
Step 3: Build the Dag
From theinclude/typescript_etl directory, build the Dag into a single file in your dags folder:
.min.mjs files that airflow-ts-pack created.
Step 4: Run the Dag
Restart your Airflow environment to install Node.js. After the Dag processor has read the file, thetypescript_etl Dag appears in the Airflow UI. Trigger a Dag run. The logs of the load task show Total order amount: 100.
Python Dags with stub tasks
You can also write the Dag in Python and implement only some of its tasks in Go, Java, or TypeScript. The Python Dag declares these tasks as stub tasks, and a task handler in Go, Java, or TypeScript implements each of them. You can mix stub tasks and regular Python tasks in the same Dag, and stub tasks of different languages too. The following example implements each task of the example Dag in a different language:extract in Go, transform in Java, and load in TypeScript.
Step 1: Configure all three coordinators
Configure the coordinators of all three languages with the following two environment variables. When you run Airflow locally, add them to your.env file and restart Airflow with astro dev restart. On Astro, add them in the Astro Environment Manager. The Java and TypeScript runtimes from the earlier sections also need to be installed.
Step 2: Write the Python Dag
Create a file calledpolyglot_etl.py in your dags folder:
queue of each stub task routes it to the coordinator of its language, and the task id is the name of the function. The parameters of the stub function define the arguments that the task handler receives. Like in the Python version of the Dag, load(transform(extract())) sets the dependencies and passes the return value of each task to the next one as an XCom.
Step 3: Write the task handlers
A task handler contains thedag_id and task_id of a stub task and the task function. Each language needs its own project for its task handler.
-
Create a Go project in
include/polyglot_gowith thego.modfile from Step 2 of the Go example. Then create amain.gofile with the task handler of theextracttask: -
Create a Java project in
include/polyglot_javawith the Gradle files from Step 2 of the Java example, and setmainClass = "com.example.etl.StubHandlers"inbuild.gradle. Then create the task handler of thetransformtask atsrc/main/java/com/example/etl/StubHandlers.java: -
Create a TypeScript project in
include/polyglot_typescriptwith the commands from Step 2 of the TypeScript example. Then create the task handler of theloadtask atsrc/polyglot_load.ts:The TypeScript task handler receives its arguments as one object, with the names of the parameters of the stub function.
Step 4: Build and run the Dag
- Build each project with the build commands from Step 3 of the Go, Java, and TypeScript examples. Give the built files new names in your
dagsfolder, for examplepolyglot_extract,polyglot_transform.jar, andpolyglot_load.min.mjs. - Trigger the
polyglot_etlDag. The logs of theloadtask showTotal order amount: 100, written by the TypeScript task handler.
On Astro, create a worker queue for each queue in
AIRFLOW__SDK__QUEUE_TO_COORDINATOR. For this example, you need the worker queues golang, java, and typescript.YAML Dags
In Airflow 3.4, you can also write Dags as YAML files. The tasks of a YAML Dag either use an existing operator, or run code from a task handler written with a language SDK. YAML Dags are experimental and still under development. See YAML Dags in the Airflow documentation.Other ways to run tasks in other languages
You can also run tasks in other languages using the following methods:- Use the
BashOperatorto run a script in another language. For example, you can use theBashOperatorto run a JavaScript or R script. See Run a script in another programming language for more information. - Use the
KubernetesPodOperatorto run any Docker image, which can include code in any language. See Use theKubernetesPodOperatorto run a script in another language for more information.