>
WEBINARS

What’s New in Airflow 2.3

Watch Video On Demand

Hosted By

  • Kenten Danas
  • Viraj Parekh

Useful links:

1. What is Apache Airflow?

Apache Airflow is one of the world’s most popular data orchestration tools — an open-source platform that lets you programmatically author, schedule, and monitor your data pipelines.

Apache Airflow was created by Maxime Beauchemin in late 2014, and brought into the Apache Software Foundation’s Incubator Program two years later. In 2019, Airflow was announced as a Top-Level Apache Project, and it is now considered the industry’s leading workflow orchestration solution.

Key benefits of Airflow:

2. Airflow Updates Timeline

whats-new-airflow-2-3-image2

3. What’s new in Airflow 2.3?

4. Major New Features

Dynamic Task Mapping

Dynamic task mapping allows DAGs to take a set of inputs at run time and creates a single task for each one.

This is helpful for use cases like:

Benefits of Dynamic Task Mapping:

Dynamic Task Mapping is simple to implement with two new functions:

The following task code example uses both of these functions to dynamically generate 3 task runs:

@task
    def add(x: int, y: int):
        return x + y

    added_values = add.partial(y=10).expand(x=[1, 2, 3])

Grid View

The grid view is a UI update replacing the tree view, focusing on task history. The grid view shows runs and tasks but leaves dependency lines to the graph view, and handles task groups better.

Example of an ELT task grid view:

whats-new-airflow-2-3-image1

Benefits of the grid view:

5. Demo

Begins at 15:15 in the webinar video.

Demo example: A task called “make_list” makes a list to map over, returning a list of some random number of integers between two and four. That list then gets passed to the downstream task that is going to take it and get mapped accordingly.

whats-new-airflow-2-3-image4

The graph view of two tasks:

whats-new-airflow-2-3-image3

More examples discussed in the Demo and Q&A.

6. Minor New Features

New/Updated Operators

Connection Updates

Other great additions

7. Learn more

Ready to Get Started?

Get Started Free

Try Astro free for 14 days and power your next big data project.