TaskFlow API in Airflow 2.0

WATCH ON DEMAND

Summary:

Learn how the TaskFlow API in Airflow 2.0 enables a better DAG authoring experience.

In this webinar we covered:

  • What is the TaskFlow API
  • TaskFlow API Features
  • Decorators
  • Custom Xcom Backends
  • Future Work

Missed the Webinar? Sign up for the Recap

Recap Preview

What is the TaskFlow API?

Prior to Airflow 2.0, Airflow did not have an explicit way to declare messages passed between tasks in a DAG.

XComs could be used, but were hidden in execution functions inside the operator.

The TaskFlow API is a functional API that allows you to explicitly declare message passing while implicitly declaring task dependencies.

TaskFlow API Features

TaskFlow API Functionality Includes:

  • XCom Args, which allow task dependencies to be abstracted and inferred as a result of the Python function invocation
  • A task decorator that automatically creates PythonOperator tasks from Python functions and handles variable passing
  • Support for Custom XCom Backends

Hosted By

Viraj Parekh

Viraj Parekh

Field CTO