From Operators to DagRuns
How Work Gets Executed
Operators become Tasks
- Operators contain the logic - but once they are added to a DAG file with a
dag, they become tasks.
- Being explicit, when an operator class is instantiated with a
dag(along with its other settings) it becomes a task within a DAG.
Tasks become Task Instances
- Once a series of tasks becomes bundled to the same DAG object, the DAG can be executed based on its schedule.
- The scheduler "taps" the DAG and begins to execute the tasks depending on their dependencies.
- Tasks that get executed have a
execution_dateand are now called task instances. These get logged in the metadata database.
DAGs become DAG Runs
- DAGs that have run or are running (i.e. have an associated
execution_date) are referred to as DAG Runs.
- DAG Runs are logged in the metadata database with their corresponding states.
- Tasks associated with a DAG Run are called task instances.
A Dag Run is an instantiation of a DAG object in time.
A Task Instance is an instantiation of a Task in time and in a DAG object.
States are used to keep track of how scheduled tasks and DAG Runs are doing. DAG Runs and tasks can have the following states:
Ready to run production-grade Airflow?
Astronomer is the easiest way to run Apache Airflow. Choose from a fully hosted Cloud option or an in-house Enterprise option and run a production-grade Airflow stack, including monitoring, logging, and first-class support.