Is it just me or was that the shortest month ever? July flew by for the Airflow Community, and we have a lot to show for it! Including:

• The release of Airflow 2.9.3

• The impending deadline to book your ticket to Airflow Summit

• And the announcement of two new committers, Rom Sharon & Shahar Epstein!

As always, please reach out with items for the next issue.

Thanks for reading!

2 Weeks Left to Book Your Hotel!

With Airflow Summit just around the corner (Sept. 10th-12th), the deadline to book your hotel room at a discounted rate is fast approaching! To secure your spot, book your hotel by August 12th. 

And, make sure to secure your tickets to celebrate 10 years of Airflow this fall!

The next Airflow Monthly Virtual Town Hall will be held on Wednesday, August 7th. We’ll receive demos on impactful PRs made in the past month and get a deep dive on what to expect in the 2.10 release. Please register here if you haven’t already.

Airflow 2.9.3 was released on July 16th, and includes several improvements and relevant bug fixes.

• PyPI | Release Notes | Docs | Sources | Docker | Constraints

Providers

New versions of 53 Airflow Providers packages were released.

For all source releases, PyPI packages and docs, see: 

• Sources | PyPI | Docs

Ecosystem

Recent releases: 

• Astro Platform Release

• OpenLineage 1.19.0

• Cosmos 1.5.1

What’s Coming in Airflow 2.10

Here are some of the exciting features you can look forward to in the 2.10 release:

• Task Instance Try History: This enables them to see previous tries of task instances.

• Hybrid Execution: Which enables the use of multiple executors for different tasks in Airflow

• Start Execution from Triggerer: Will be extended to include mapped task instances

• Dataset UI: Will receive some improvements

• 8/7 Meetup in LA: Los Angeles Apache Airflow Chapter Launch!

• 8/7 Webinar: Airflow Monthly Town Hall

• 8/8 Webinar: What’s New in the Astro Platform Release, Q2 2024

• 8/15 Webinar: Modern Infrastructure for World Class AI Applications

• 8/22 Meetup in Palo Alto: Apache Airflow Meets Generative AI

• 9/10-12 Conference in San Francisco: Airflow Summit 2024

• 9/11 Event in San Francisco: Airflow Summit After Party

• 11/19-21 Virtual Conference: The Open Source Analytics Conference

• 7/9 Webinar: Best Practices for Effectively Running dbt in Airflow

• 7/10 Webinar: Airflow Monthly Town Hall

• 7/17 Meetup in NYC: Airflow Rooftop Happy Hour

• 7/24 Meetup in London: Data Engineer Thing SUMMER BASH!

Read

• Airflow + Bank of Canada API for daily foreign exchange rates by Mihir Samant on Airflow Medium

• Astronomer Adopts DAG Factory to Democratize Writing Data Pipelines by Viraj Parekh & Julian LaNeve on the Astronomer Blog

• Airflow Installation Simplified — using Docker Compose/Podman Compose by Raviteja Tholupunoori on Airflow Medium

• What’s New in the Astro Platform Release by Radhika Gulati on the Astronomer Blog

• Solving Data Pipeline Challenges with Apache Airflow: A Real-Life Example by Raviteja Tholupunoori on Airflow Medium

• Airflow and dbt: the next chapter by Julian LaNeve on the Astronomer Blog

• How Laurel Uses Airflow to Enhance Machine Learning Pipelines with Vincent La and Jim Howard on The Data Flowcast

• Airflow Strategies for Business Efficiency at Campbell with Larry Komenda on The Data Flowcast

   

• Best Practices for Effectively Running dbt in Airflow on Astronomer Webinars

• How to Orchestrate Databricks Jobs Using Airflow on Astronomer Webinars

Votes: 

• Switching to trusted publishing workflow for PyPI

• Airflow Providers prepared on July 02, 2024 (openlineage rc2)

• [RESULT][VOTE] Airflow Providers - release prepared on July 02, 2024

• Airflow Providers prepared on July 09, 2024

• [RESULT][VOTE] Airflow Providers - release of July 09, 2024

• Airflow Providers prepared on July 12, 2024

• [RESULT][VOTE] Airflow Providers - release of July 12, 2024

• Release Airflow 2.9.3 from 2.9.3rc1

• [RESULT][VOTE] Release Airflow 2.9.3 from 2.9.3rc1

• AIP-72: Task Execution Interface aka Task SDK

• [RESULT][VOTE] Accepted - AIP-72: Task Execution Interface aka Task SDK

• Release Apache Airflow Helm Chart 1.15.0 based on 1.15.0rc1

• AIP-66: DAG Bundles and Parsing

• (v2) AIP-69 Remote Executor

• Release Apache Airflow Python Client 2.9.3 from 2.9.3rc1

• Airflow Providers prepared on July 21, 2024

• [RESULT][VOTE] Airflow Providers - release of July 21, 2024

• July 2024 PR of the Month

• AIPs 73 74 75: Data Awareness with Assets

• [RESULT][VOTE] AIPs 73 74 75: Data Awareness with Assets

• Release Apache Airflow Python Client 2.9.1 from 2.9.1rc1

• [RESULT][VOTE] Release Apache Airflow Python Client 2.9.1 from 2.9.1rc1

• AIP-80: Explicit Template Fields in Operator Arguments

• AIP-68 Extended Plugin Interface for React Views

• AIP-78 Scheduler-managed backfills

• AIP-78 Scheduler-managed backfills

• [RESULT][VOTE] AIP-78 Scheduler-managed backfills

• Airflow Providers prepared on July 28, 2024

• [RESULT][VOTE] Airflow Providers - release of July 28, 2024

• [RESULT][VOTE] AIP-68 Extended Plugin Interface for React Views

• [RESULT][VOTE] July 2024 PR of the Month

Discussed:

• Broken details link in 1.10.8 release notes

• @remove provide_bucket_name and @unify_bucket_name_and_key?

• ASF download link broken

• AIP-78 scheduler-managed backfill

• Scheduler "workstream"? was: [DISCUSS] AIP-78 scheduler-managed backfill

• AIP-80: Explicit Template Fields in Operator Arguments

• AIP-66: DAG Bundles & Parsing

• AIPs 73 74 & 75: Data Awareness with Assets

• To AIP-44 or not to AIP-44

• More ruff style rules

• AIP-81: Enhanced Security in CLI via Integration of API

• Approaches to bugfixes for 2.10 after main becomes 3.0

• AIP-67 Multi-team deployment of Airflow components (reloaded)

• Add the Microsoft Fabric Provider in Apache Airflow

• External event driven scheduling in Airflow

• AIP-83 Remove Execution Date Unique Constraint from DAG Run

• Extendable parsing controls (a.k.a. Pluggable Parsers)

• AIP-76 Asset Partitions

• Proposal to Add OTEL Provider to Apache Airflow

• AIP-78 & AIP 84 Remove FAB as core dependency and UI REST API

Announced:

• Apache Airflow Providers prepared on July 02, 2024 are released

• Airflow 3 Dev call on 11 July (Thursday) with a Proposed Agenda

• Minutes and Recording from June 2024 Airflow Monthly Town Hall

• Apache Airflow Providers prepared on July 09, 2024 are released

• Apache Airflow registered as trademark in US

• Apache Airflow Providers prepared on July 12, 2024 are released

• Opportunity to Contribute to Airflow 3: Own a Workstream!

• Apache Airflow 2.9.3 Released

• Airflow 3 Dev call - 11 July 2024

• Airflow Summit 2024 - Program is ready

• Apache Airflow Helm Chart version 1.15.0 Released

• Apache Airflow Providers prepared on July 21, 2024 are released

• Apache Airflow 2.10.0b1 available for testing

• Airflow Summit 2024 Workshops

• Airflow 3 Dev call - 25 July 2024

• New committers: Rom Sharon & Shahar Epstein

• AWS Open Source Credits Program

• Apache Airflow Python Client 2.9.1 Released

• Apache Airflow Providers prepared on July 28, 2024 are released

Proposed:

• Auto-labelling Airflow issues with DoSu AI - let's teach the bot a bit

• Operator Templating in Airflow 3

• Agree and document Ad-hoc release process for providers

• AIP-44: Airflow Internal API

• AIP-49: Open Telemetry Support

• AIP-51 Removing Executor Coupling from Core Airflow

• AIP-57: Refactor SLA feature

• AIP-58: Airflow ObjectStore (AS)

• AIP-59: Performance test framework

• AIP-61 Hybrid Execution

• AIP-62 Getting Lineage from Hook Instrumentation

• AIP-63: DAG Versioning

• AIP-64: Keep TaskInstance try history

• AIP-65: Improve DAG history in UI

• AIP-66: Execution of specific DAG versions

• AIP-67 Multi-tenant deployment of Airflow components

• AIP-68 Extended Plugin Interface for Custom Grid View Panels

• AIP-69 Remote Executor

• AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

• AIP-72 Task Execution Interface aka Task SDK

New!

• AIP-73 Expanded Data Awareness

• AIP-78 Scheduler-managed backfill

• AIP-80 Explicit Template Fields in Operator Arguments

• AIP-81 Enhanced Security in CLI via Integration of API

• AIP-82 External event driven scheduling in Airflow

• AIP-83 Remove Execution Date Unique Constraint from DAG Run

Did you know that football teams wearing red kits play better? The color of your clothes can affect how you’re perceived by others and change how you feel. A review of football matches in the last 55 years, for example, showed that teams wearing a red kit consistently played better in home matches than teams in any other colour.

Good thing the Airflow logo has red in it ;).

Source: BBC

Please subscribe to this newsletter to ensure you always receive the latest issue:  http://eepurl.com/hXUA3r.

Know of an upcoming event or publication the community should know about? Let us know at briana.okyere@astronomer.io.