In November, the Airflow community was busy finalizing the release of Airflow 2.8 (TBA), in addition to enjoying themselves at AWS re:Invent, and, as always, helping each other in our Airflow journey’s on Slack, Stack, and anywhere else builders live! Additionally this month, we:

• Released Airflow 2.7.3

• Welcomed new committer Jens Scheffler 

• Hosted events in London, Cincinnati, and New York!

As always, please reach out with items for the next issue.

Thanks for reading!

Airflow 2.7.3

Airflow 2.7.3 was released on November 5th and contains several bug fixes and relevant feature updates.

• PyPI | Release Notes | Docs | Sources | Docker | Constraints

Providers

New versions of 25 Airflow Providers packages were released.

For all source releases, PyPI packages and docs, see: 

• Sources | PyPI | Docs

Ecosystem

Initial releases: 

• Astronomer Cosmos 1.2.5

• OpenLineage 1.5.0

   

• 12/7 Webinar: What’s New in the Astro Platform Release, Q4 2023

• 12/7 Meetup in Prague: First Airflow Prague Meetup with Kiwi.com & Astronomer

• 12/12 Meetup in Hyderabad: Hyderabad Airflow Meetup @ Microsoft HQ

• 12/14 Webinar: Orchestrate next-generation AI tools: Explore six new Airflow providers

• 11/2 Webinar: Run LLMOps in Production with Airflow

• 11/9 Webinar: Intro to Airflow: Get Started Writing Pipelines for Any Use Case

• 11/15 Meetup in NYC: Airflow AI Meetup

• 11/16 Meetup in Cincinnati: ML Ops in Airflow @ Astronomer HQ

• 11/16 Workshop: Get Apache Airflow® Certified For Free!

• 11/22 Meetup in London: Airflow Meetup @ Monzo

• 11/27-30th Conference in Las Vegas, NV: AWS re:Invent

• 11/29 Webinar: Deliver Data with Precision: Supercharge your Airflows with Apache Airflow® on Astro - An Azure Native ISV Service

• 11/29 Meetup in Amsterdam: Airflow Adventures: Streamlining Demand Forecasting & Operationalizing LLM Apps

• 11/30 Webinar: Airflow at Faire: Democratizing Machine Learning at Scale

• 12/1 Webinar: #AzureHappyHours- Boost Airflows: Apache on Astro- Azure ISV

• Customizable feature using Airflow Plugins by Giseung Beak on Airflow Medium

• Apache Airflow® TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs by George Yates on the Astronomer Blog

• The ins and outs of Airflow’s new Secrets Cache by Raphaël Vandon on Airflow Medium

• Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider by Nick Acosta on the Astronomer Blog

• Test Airflow Upgrades with the Astro CLI by David Koenitzer on the Astronomer Blog

• Data Engineering with Apache Airflow®, Snowflake, Snowpark, dbt & Cosmos by George Yates in Quickstart Guides

• Introducing Apache Airflow® on Astro – an Azure Native ISV Service by Conor Beverland on the Astronomer Blog

• Sensor Approach to Airflow Pipelines by Franco Nyawanga on Airflow Medium

• Call paginated API with Airflow’s HttpOperator by Joffrey Bienvenu on Airflow Medium

• Data Engineering with Apache Airflow®, Snowflake, Snowpark, dbt & Cosmos by George Yates in Quickstart Guides

• LIVE with Astronomer: Using the new Weaviate Provider for Sentiment Analysis with Airflow on Youtube

• Airflow in action: providing environmental forecast services by Eloi Codina-Torras on Airflow Medium

• LIVE with Astronomer: Airflow and Databricks, Better Together! On Youtube

• Advanced Tips & Tricks for Improving Your Airflow DAGs! On Youtube

• Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools by Steven Hillion on the Astronomer Blog

Votes: 

• [RESULT][VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

• Release Airflow 2.7.3 from 2.7.3rc1

• [RESULT][VOTE] Release Airflow 2.7.3 from 2.7.3rc1

• Agree on the process of removal of providers

• Migrating our reference images to Debian Bookworm for Airflow 2.8

• Airflow Providers prepared on November 08, 2023

• [RESULT][VOTE] Airflow Providers - release of November 08, 2023

• Decouple Core and API Clients versioning

• Release Airflow Python Client 2.7.3 from 2.7.3rc1

• [RESULT][VOTE] Release Airflow Python Client 2.7.3 from 2.7.3rc1

• Airflow Providers prepared on November 12, 2023

• [RESULT][VOTE] Airflow Providers - release of November 12, 2023

• Remove Qubole provider

• Airflow Providers prepared on November 24, 2023

• Add fab provider

• [RESULT][VOTE] Airflow Providers - release of November 24, 2023

• November PR of the Month

• Remove 'daskexecutor' provider

• Change default docker image to point to "latest supported"

Discussed:

• Requesting Feedback on Airflow Survey

• API Clients Major version

• Airflow UI DAG Composer

• Move FAB auth manager to a new provider

• Suspend (Remove?) Daskexecutor provider

• Suspend/Remove Apache Scoop provider

• Suspend/Remove Plexus provider

Announced:

• Apache Airflow® 2.7.3 Released

• How to reproducibly install Airflow

• The 2023 Airflow Survey is LIVE

• New committer: Jens Scheffler

• Initiative on Strengthening security for Apache Airflow®

• Apache Airflow® Providers prepared on November 08, 2023 are released

• Apache Airflow® Python Client 2.7.3 Released

• Apache Airflow® Providers prepared on November 12, 2023 are released

• Apache Airflow® 2.8.0b1 available for testing

• Apache Airflow® Providers prepared on November 24, 2023 are released

• Change default docker image to point to "latest supported"

• Deprecate URI Connection representation in favor of JSON

• Airflow Monthly Town-Hall

• AIP-51 Removing Executor Coupling from Core Airflow

• AIP-56: Extensible user management

• AIP-58 Airflow ObjectStore (AS)

• See the Wiki for more of the project’s approved AIPs.

 

Did you know that Earth’s rotation is changing speed. It's actually slowing. This means that, on average, the length of a day increases by around 1.8 seconds per century. 600 million years ago a day lasted just 21 hours.

Please subscribe to this newsletter to ensure you always receive the latest issue:  http://eepurl.com/hXUA3r.

Know of an upcoming event or publication the community should know about? Let us know at viraj@astronomer.io or briana.okyere@astronomer.io.