Create and use params in Airflow
Params are arguments which you can pass to an Airflow DAG or task at runtime and are stored in the Airflow context dictionary for each DAG run. You can pass DAG and task-level params by using the params
parameter.
Params are ideal to store information that is specific to individual DAG runs like changing dates, file paths or ML model configurations. Params are not encrypted and therefore not suitable to pass secrets. See also Best practices for storing information in Airflow.
This guide covers:
- How to pass params to a DAG at runtime.
- How to define DAG-level param defaults which are rendered in the Trigger DAG UI.
- How to access params in an Airflow task.
- The hierarchy of params in Airflow.
Assumed knowledge
To get the most out of this guide, you should have an understanding of:
- Airflow DAGs. See Introduction to Airflow DAGs.
- Airflow operators. See Operators 101.
- Airflow context. See Access the Apache Airflow context.
Pass params to a DAG run at runtime
Params can be passed to a DAG at runtime in four different ways:
- In the Airflow UI by using the Trigger DAG form. This form appears when you click on the Trigger DAG (Play) button in the Airflow UI.
- Running a DAG with the
--conf
flag using the Airflow CLI (airflow dags trigger
). - Using the TriggerDagRunOperator with the
conf
parameter. - Making a
POST
request to the Airflow REST APIs Trigger Dag Run endpoint and using theconf
parameter.
Param values passed to a DAG by any of these methods will override existing default values for the same key as long as the Airflow core config dag_run_conf_overrides_params
is set to True
(which is the default setting).
You can only pass params to a DAG that are JSON-serializeable. Any params that are not JSON-serializeable will cause a DAG Import Error (ParamValidationError
) upon DAG parsing.
Trigger DAG form
You can pass params to DAGs from the Airflow UI by clicking on the Play button in the DAGs overview or on the blue Trigger button on an individual DAG page.
This button opens a form in which you can specify details for the DAG run. If the DAG has any params with defaults values defined, the form will render a field for each those params under Run Parameters. Additional params can be added under Advanced Options -> Configuration JSON.
Under Advanced Options, you can also set the Logical Date, define a Run ID, and add a Dag Run Note for the manual DAG run. Note that any params already defined under Run Parameters will be included in the Configuration JSON by default.
In the Trigger DAG form:
- You can change any default values for params that have been defined in the DAG file under Run Parameters or in the Configuration JSON.
- You can add new params under Advanced Options -> Configuration JSON.
- You can set the Logical Date of the DAG run to any date. Note that you can also provide no logical date, by clearning it in the calendar picker.
- You can set the Run ID to any string. If no Run ID is specified, Airflow generates one based on the run after date.
- You can add a Dag Run Note to the DAG run.
After setting the configuration, you can start the DAG run with the Trigger button.
CLI
When you run an Airflow DAG from the CLI, you can pass params to the DAG run by providing a JSON string to the --conf
flag. For example, to trigger the params_default_example
DAG with the value of Hello from the CLI
for param1
, run:
- Astro CLI
- Airflow CLI
Run Airflow commands from the Astro CLI using astro dev run
:
astro dev run dags trigger params_defaults_example --conf '{"param1" : "Hello from the CLI"}'
airflow dags trigger params_defaults_example --conf '{"param1" : "Hello from the CLI"}'
The CLI prints the configuration for the triggered run to the command line:
You can use a --conf
flag with the following Airflow CLI sub-commands:
airflow dags backfill
airflow dags test
airflow dags trigger
TriggerDagRunOperator
The TriggerDagRunOperator is a core Airflow operator that allows you to start a DAG run from within another DAG. You can use the TriggerDAGRunOperator conf
param to trigger the dependent DAG with a specific configuration.
The DAG below uses the TriggerDagRunOperator to trigger the tdro_example_downstream
DAG while passing a dynamic value for the upstream_color
param via the conf
parameter. The value for upstream_color
is passed via a Jinja template pulling the return value of an upstream task via XCom.
- TaskFlow
- Traditional syntax
from airflow.sdk import dag, task, chain
from airflow.providers.standard.operators.trigger_dagrun import TriggerDagRunOperator
import random
@dag
def tdro_example_upstream():
@task
def choose_color():
color = random.choice(["blue", "red", "green", "yellow"])
return color
tdro = TriggerDagRunOperator(
task_id="tdro",
trigger_dag_id="tdro_example_downstream",
conf={"upstream_color": "{{ ti.xcom_pull(task_ids='choose_color')}}"},
)
chain(choose_color(), tdro)
tdro_example_upstream()
from airflow.sdk import dag, chain
from airflow.providers.standard.operators.trigger_dagrun import TriggerDagRunOperator
from airflow.providers.standard.operators.python import PythonOperator
import random
def choose_color_func():
color = random.choice(["blue", "red", "green", "yellow"])
return color
@dag
def tdro_example_upstream_traditional():
choose_color = PythonOperator(
task_id="choose_color",
python_callable=choose_color_func,
)
tdro = TriggerDagRunOperator(
task_id="tdro",
trigger_dag_id="tdro_example_downstream",
conf={"upstream_color": "{{ ti.xcom_pull(task_ids='choose_color')}}"},
)
chain(choose_color, tdro)
tdro_example_upstream_traditional()
Runs of the tdro_example_downstream
DAG that are triggered by this upstream DAG will override the default value of the upstream_color
param with the value passed via the conf
parameter, which leads to the print_color
task to print either red
, green
, blue
or yellow
.
- TaskFlow
- Traditional syntax
from airflow.sdk import dag, task
@dag(
params={"upstream_color": "Manual run, no upstream color available."},
)
def tdro_example_downstream():
@task
def print_color(**context):
print(context["params"]["upstream_color"])
print_color()
tdro_example_downstream()
from airflow.sdk import dag
from airflow.providers.standard.operators.python import PythonOperator
def print_color_func(**context):
print(context["params"]["upstream_color"])
@dag(
params={"upstream_color": "Manual run, no upstream color available."},
)
def tdro_example_downstream_traditional():
PythonOperator(
task_id="print_color",
python_callable=print_color_func,
)
tdro_example_downstream_traditional()
Define DAG-level param defaults
To specify params for all runs of a given DAG, pass default values to the param
parameter of the @dag
decorator or the DAG
class in your DAG file. You can directly specify a default value or use the Param
class to define a default value with additional attributes.
The DAG below has two DAG-level params with defaults: param1
and param2
, the latter only accepting integers.
- TaskFlow
- Traditional syntax
from airflow.sdk import dag, task, Param
@dag(
params={
"param1": "Hello!",
"param2": Param(
23,
type="integer",
),
},
)
def simple_param_dag():
@task
def print_all_params(**context):
print(context["params"]["param1"] * 3)
print(context["params"]["param2"])
print_all_params()
simple_param_dag()
from airflow.sdk import DAG, Param
from airflow.providers.standard.operators.python import PythonOperator
def print_all_params_func(**context):
print(context["params"]["param1"] * 3)
print(context["params"]["param2"])
with DAG(
dag_id="simple_param_dag",
params={
"param1": "Hello!",
"param2": Param(
23,
type="integer",
),
},
):
PythonOperator(
task_id="print_all_params",
python_callable=print_all_params_func,
)
If you define DAG-level param defaults, the Trigger DAG form renders a field for each param. From this UI, you can then override your defaults for individual DAG runs. A param with a red asterisk is a required param.
When you specify a required type
for a param, the field will be a required input by default because of JSON validation. To make a field optional but still require a specific input type, allow NULL values by setting the type to ["null", "<my_type>"]
.
If you do not specify a type for your param, Airflow will infer it based on the default value you provide.
Param types
The following param types are supported:
string
: A string. This is the default type.null
: Allows the param to be None by being left empty.integer
: An integer.number
: A float (or integer).boolean
:True
orFalse
.array
: An HTML multi line text field, every line edited will be made into a string array as the value.object
: A JSON entry field.
Param attributes
Aside from the type
attribute, the Param
class has several other attributes that you can use to define how users interact with the param:
title
: The title of the param that appears in the Trigger DAG UI.description
: A description of the param.section
: Creates a section under which the param will appear in the Trigger DAG UI. All params with no specified section will appear under the default section DAG conf Parameters.format
: A JSON format that Airflow will validate a user's input against.enum
: A list of valid values for a param. Setting this attribute creates a dropdown menu in the UI.const
: Defines a permanent default value and hides the param from the Trigger DAG UI. Note that you still need to provide adefault
value for the param.
All Param
attributes are optional to set. For string type params, you can additionally set minLength
and maxLength
to define the minimum and maximum length of the input. Similarly, integer and number type params can have a minimum
and maximum
value.
Param examples in the Airflow UI
This section presents a few examples of params and how they are rendered in the Trigger DAG UI.
The code snippet below defines a mandatory string param with a few UI elements to help users input a value.
from airflow.sdk import Param
"my_string_param": Param(
"Airflow is awesome!",
type="string",
title="Favorite orchestrator:",
description="Enter your favorite data orchestration tool.",
section="Important params",
minLength=1,
maxLength=200,
)
When you define date, datetime, or time param, a calendar picker appears in the Trigger DAG UI.
from airflow.sdk import Param
"my_datetime_param": Param(
"2016-10-18T14:00:00+00:00",
type="string",
format="date-time",
),
Providing a list of values to the enum
attribute will create a dropdown menu in the Trigger DAG UI. Note that the default value must also be in the list of valid values provided to enum
. Due to JSON validation rules, a value has to be selected.
from airflow.sdk import Param
"my_enum_param": Param(
"Hi :)", type="string", enum=["Hola :)", "Hei :)", "Bonjour :)", "Hi :)"]
),
A boolean type param will create a toggle in the Trigger DAG UI.
from airflow.sdk import Param
"my_bool_param": Param(True, type="boolean"),
An array type param will create a multi-line text field in the Trigger DAG UI. Each line will be converted to an item in the array.
from airflow.sdk import Param
"my_array_param": Param(["Hello Airflow", ":)"], type="array"),
An object type param will create a JSON entry field in the Trigger DAG UI. The value of the param must be a valid JSON object.
from airflow.sdk import Param
"my_object_param": Param({"a": 1, "b": 2}, type="object"),
Define task-level param defaults
You can set task-level param defaults in the same way as for DAG-level params. If a param of the same key is specified at both the DAG and task level, the DAG-level param will take precedence.
- TaskFlow
- Traditional syntax
@task(params={"param1": "Hello World!"})
def t1(**context):
print(context["params"]["param1"])
t1 = BashOperator(
task_id="t1",
bash_command="echo {{ params.param1 }}",
params={"param1": "Hello World!"},
)
Access params in a task
You can access params in an Airflow task like you can with other elements in the Airflow context.
- TaskFlow
- Traditional syntax
from airflow.sdk import task
@task
def t1(**context):
print(context["params"]["my_param"])
from airflow.providers.standard.operators.python import PythonOperator
def t1_func(**context):
print(context["params"]["my_param"])
t1 = PythonOperator(
task_id="t1",
python_callable=t1_func,
)
Params are also accessible as a Jinja template using the {{ params.my_param }}
syntax.
If you try to access a param that has not been specified for a specific DAG run, the task will fail with an exception.
Param precedence
The order of precedence for params, with the first item taking most precedence, is as follows:
- Params that have been provided for a specific DAG run by a method detailed in pass params to a DAG run at runtime as long as the Airflow config core.dag_run_conf_overrides_params is set to
True
. - Param defaults that have been defined at the DAG level.
- Param defaults that have been defined at the task level.