WEBINAR

How to pass data between your Airflow tasks

Recorded On April 11, 2023

  • Tamara Fingerlin
  • Kenten Danas

This webinar provides a deep dive into passing data between your Airflow tasks. We cover everything you need to know to choose and implement a method for passing data based on your use case, infrastructure, scale, and more. Questions covered in this session include:

  • What are some best practices to follow when using XComs, Airflow’s built-in cross-communication utility?
  • How can I pass data between tasks and DAGs using TaskFlow and traditional operators?
  • How does the Astro Python SDK use XComs to move data between relational stores and Python data structures, and how can it simplify my pipelines?
  • How can I set up a custom XCom backend and implement custom serialization methods?
  • How can I pass data between tasks that are run in isolated environments like the KubernetesPodOperator?

All code covered in this webinar can be found in this repo.

Build, run, & observe your data workflows.
All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $20 in free credits during your 14-day trial.