Using the BashOperator

The BashOperator is one of the most commonly used operators in Airflow. It executes bash commands or a bash script from within your Airflow DAG.

In this guide you’ll learn:

• When to use the BashOperator.
• How to use the BashOperator and @task.bash decorator.
• How to use the BashOperator including executing bash commands and bash scripts.
• How to run scripts in non-Python programming languages using the BashOperator.

Assumed knowledge

To get the most out of this guide, you should have an understanding of:

• Airflow operators. See Operators 101.
• Airflow decorators. See Introduction to the TaskFlow API and Airflow decorators.
• Basic bash commands. See the Bash Reference Manual.

How to use the BashOperator and @task.bash decorator

The BashOperator is part of core Airflow and can be used to execute a single bash command, a set of bash commands, or a bash script ending in .sh. The @task.bash decorator can be used to create bash statements using Python functions and is available as of Airflow 2.9.

1
from airflow.providers.standard.operators.bash import BashOperator 
2
3
bash_task = BashOperator(
4
    task_id="bash_task",
5
    bash_command="echo $MY_VAR",
6
    env={"MY_VAR": "Hello World"}
7
)

1
from airflow.sdk import task
2
3
@task.bash(env={"MY_VAR": "Hello World"})
4
def bash_task():
5
    return "echo $MY_VAR"  # the returned string is executed as a bash command
6
7
bash_task()

The following parameters can be provided to the operator and decorator:

• bash_command: Defines a single bash command, a set of commands, or a bash script to execute. This parameter is required.
• env: Defines environment variables in a dictionary for the bash process. By default, the defined dictionary overwrites all existing environment variables in your Airflow environment, including those not defined in the provided dictionary. To change this behavior, you can set the append_env parameter. If you leave this parameter blank, the BashOperator inherits the environment variables from your Airflow environment.
• append_env: Changes the behavior of the env parameter. If you set this to True, the environment variables you define in env are appended to existing environment variables instead of overwriting them. The default is False.
• output_encoding: Defines the output encoding of the bash command. The default is utf-8.
• skip_on_exit_code: Defines which bash exit code should cause the BashOperator to enter a skipped state. The default is 99.
• cwd: Changes the working directory where the bash command is run. The default is None and the bash command runs in a temporary directory.

The behavior of a BashOperator task is based on the status of the bash shell:

• Tasks succeed if the whole shell exits with an exit code of 0.
• Tasks are skipped if the exit code is 99 (unless otherwise specified in skip_exit_code).
• Tasks fail in case of all other exit codes.

Tip

If you expect a non-zero exit from a sub-command you can add the prefix set -e; to your bash command to make sure that the exit is captured as a task failure.

Both the bash_command and the env parameter can accept Jinja templates. However, the input given through Jinja templates to bash_command is not escaped or sanitized. If you are concerned about potentially harmful user input you can use the setup shown in the BashOperator documentation.

When to use the BashOperator

The following are common use cases for the BashOperator and @task.bash decorator in Airflow DAGs:

• Creating and running bash commands based on complex Python logic.
• Running a single or multiple bash commands in your Airflow environment.
• Running a previously prepared bash script.
• Running scripts in a programming language other than Python.
• Running commands to initialize tools that lack specific operator support. For example Soda Core.

Example: Using Python to create bash commands

You can use @task.bash to create bash statements using Python functions. This decorator is especially useful when you want to run bash commands based on complex Python logic, including inputs from upstream tasks. The following example demonstrates how to use the @task.bash decorator to conditionally run different bash commands based on the output of an upstream task.

1
from airflow.sdk import task
2
3
@task
4
def upstream_task():
5
    dog_owner_data = {
6
        "names": ["Trevor", "Grant", "Marcy", "Carly", "Philip"],
7
        "dogs": [1, 2, 2, 0, 4],
8
    }
9
10
    return dog_owner_data
11
12
@task.bash
13
def bash_task(dog_owner_data):
14
    names_of_dogless_people = []
15
    for name, dog in zip(dog_owner_data["names"], dog_owner_data["dogs"]):
16
        if dog < 1:
17
            names_of_dogless_people.append(name)
18
19
    if names_of_dogless_people:
20
        if len(names_of_dogless_people) == 1:
21
            # this bash command is executed if only one person has no dog
22
            return f'echo "{names_of_dogless_people[0]} urgently needs a dog!"'
23
        else:
24
            names_of_dogless_people_str = " and ".join(names_of_dogless_people)
25
            # this bash command is executed if more than one person has no dog
26
            return f'echo "{names_of_dogless_people_str} urgently need a dog!"'
27
    else:
28
        # this bash command is executed if everyone has at least one dog
29
        return f'echo "All good, everyone has at least one dog!"'
30
31
bash_task(dog_owner_data=upstream_task())

Example: Execute two bash commands using one BashOperator

The BashOperator can execute any number of bash commands separated by &&.

In this example, you run two bash commands in a single task:

• echo Hello $MY_NAME! prints the environment variable MY_NAME to the console.
• echo $A_LARGE_NUMBER | rev 2>&1 | tee $AIRFLOW_HOME/include/my_secret_number.txt takes the environment variable A_LARGE_NUMBER, pipes it to the rev command which reverses any input, and saves the result in a file called my_secret_number.txt located in the /include directory. The reversed number will also be printed to the console.

The second command uses an environment variable from the Airflow environment, AIRFLOW_HOME. This is only possible because append_env is set to True.

1
from airflow.decorators import dag
2
from airflow.operators.bash import BashOperator
3
from pendulum import datetime
4
5
6
@dag(start_date=datetime(2022, 8, 1), schedule=None, catchup=False)
7
def bash_two_commands_example_dag():
8
    say_hello_and_create_a_secret_number = BashOperator(
9
        task_id="say_hello_and_create_a_secret_number",
10
        bash_command="echo Hello $MY_NAME! && echo $A_LARGE_NUMBER | rev  2>&1\
11
                     | tee $AIRFLOW_HOME/include/my_secret_number.txt",
12
        env={"MY_NAME": "<my name>", "A_LARGE_NUMBER": "231942"},
13
        append_env=True,
14
    )
15
16
    say_hello_and_create_a_secret_number
17
18
19
bash_two_commands_example_dag()

It is also possible to use two separate BashOperators to run the two commands, which can be useful if you want to assign different dependencies to the tasks.

Example: Execute a bash script

The BashOperator can also be provided with a bash script (ending in .sh) to be executed.

For this example, you run a bash script which iterates over all files in the /include folder and prints their names to the console.

$
#!/bin/bash
>
>
echo "The script is starting!"
>
echo "The current user is $(whoami)"
>
files = $AIRFLOW_HOME/include/*
>
>
for file in $files
>
do
>
    echo "The include folder contains $(basename $file)"
>
done
>
>
echo "The script has run. Have an amazing day!"

Make sure that your bash script (my_bash_script.sh in this example) is available to your Airflow environment. If you use the Astro CLI, you can make this file accessible to Airflow by placing it in the /include directory of your Astro project.

It is important to make the bash script executable by running the following command before making the script available to your Airflow environment:

$
chmod +x my_bash_script.sh

If you use the Astro CLI, you can run this command before running astro dev start, or you can add the command to your project’s Dockerfile with the following RUN command:

1
RUN chmod +x /usr/local/airflow/include/my_bash_script.sh

Astronomer recommends running this command in your Dockerfile for production builds such as Astro Deployments or in production CI/CD pipelines.

After making the script available to Airflow, you only have to provide the path to the script in the bash_command parameter. Be sure to add a space character at the end of the filepath, or else the task will fail with a Jinja exception!

1
from airflow.decorators import dag
2
from airflow.operators.bash import BashOperator
3
from pendulum import datetime
4
5
6
@dag(start_date=datetime(2022, 8, 1), schedule=None, catchup=False)
7
def bash_script_example_dag():
8
    execute_my_script = BashOperator(
9
        task_id="execute_my_script",
10
        # Note the space at the end of the command!
11
        bash_command="$AIRFLOW_HOME/include/my_bash_script.sh ",
12
        # since the env argument is not specified, this instance of the
13
        # BashOperator has access to the environment variables of the Airflow
14
        # instance like AIRFLOW_HOME
15
    )
16
17
    execute_my_script
18
19
20
bash_script_example_dag()

Example: Run a script in another programming language

Using the BashOperator is a straightforward way to run a script in a non-Python programming language in Airflow. You can run a script in any language that can be run with a bash command.

In this example, you run some JavaScript to query a public API providing the current location of the international Space Station. The query result is pushed to XCom so that a second task can extract the latitude and longitude information in a script written in R and print the data to the console.

The following setup is required:

• Install the JavaScript and R language packages at the OS level.
• Write a JavaScript file.
• Write a R script file.
• Make the scripts available to the Airflow environment.
• Execute the files from within a DAG using the BashOperator.

If you use the Astro CLI, the programming language packages can be installed at the OS level by adding them to the packages.txt file of your Astro project.

r-base
nodejs

The following JavaScript file contains code for sending a GET request to the /iss-now path at api.open-notify.org and returning the results to stdout, which will both be printed to the console and pushed to XCom by the BashOperator.

1
// specify that a http API is queried
2
const http = require('http');
3
4
// define the API to query
5
const options = {
6
  hostname: 'api.open-notify.org',
7
  port: 80,
8
  path: '/iss-now',
9
  method: 'GET',
10
};
11
12
const req = http.request(options, res => {
13
  // log the status code of the API response
14
  console.log(`statusCode: ${res.statusCode}`);
15
16
  // write the result of the GET request to stdout
17
  res.on('data', d => {
18
    process.stdout.write(d);
19
  });
20
});
21
22
// in case of an error print the error statement to the console
23
req.on('error', error => {
24
  console.error(error);
25
});
26
27
req.end();

The second task runs a script written in R that uses a regex to filter and print the longitude and latitude information from the API response.

1
# Print outputs to the console
2
options(echo = TRUE)
3
4
# Read all trailing command line args 
5
myargs <- commandArgs(trailingOnly = TRUE)
6
7
# Reassemble them into a single JSON string
8
json_string <- paste(myargs, collapse = " ")
9
10
# Use regex to extract latitude & longitude 
11
latitude_str <- sub('.*"latitude"\\s*:\\s*"([^"]+)".*', '\\1', json_string)
12
longitude_str <- sub('.*"longitude"\\s*:\\s*"([^"]+)".*', '\\1', json_string)
13
14
latitude <- as.numeric(latitude_str)
15
longitude <- as.numeric(longitude_str)
16
17
sprintf("The current ISS location: lat: %s / long: %s.", latitude, longitude)

To run these scripts using the BashOperator, ensure that they are accessible to your Airflow environment. If you use the Astro CLI, you can place these files in the /include directory of your Astro project.

The DAG uses the BashOperator to execute both files defined above sequentially.

1
from airflow.decorators import dag
2
from airflow.operators.bash import BashOperator
3
from pendulum import datetime
4
5
6
@dag(
7
    dag_id="print_ISS_info_dag",
8
    start_date=datetime(2022, 8, 1),
9
    schedule=None,
10
    catchup=False,
11
)
12
def print_ISS_info_dag():
13
    # Use the node command to execute the JavaScript file from the command line
14
    get_ISS_coordinates = BashOperator(
15
        task_id="get_ISS_coordinates",
16
        bash_command="node $AIRFLOW_HOME/include/my_java_script.js",
17
    )
18
19
    # Use the Rscript command to execute the R file which is being provided
20
    # with the result from task one via an environment variable via XComs
21
    print_ISS_coordinates = BashOperator(
22
        task_id="print_ISS_coordinates",
23
        bash_command="Rscript $AIRFLOW_HOME/include/my_R_script.R $ISS_COORDINATES",
24
        env={
25
            "ISS_COORDINATES": "{{ task_instance.xcom_pull(\
26
                               task_ids='get_ISS_coordinates', \
27
                               key='return_value') }}"
28
        },
29
        # set append_env to True to be able to use env variables
30
        # like AIRFLOW_HOME from the Airflow environment
31
        append_env=True,
32
    )
33
34
    get_ISS_coordinates >> print_ISS_coordinates
35
36
37
print_ISS_info_dag()