globe
Webinars

Running Airflow Tasks in Isolated Environments

Watch On Demand

Missed the Webinar? Watch Now.

By proceeding you agree to our Privacy Policy , our Website Terms and to receive emails from Astronomer.

Summary

The ability to run tasks in an isolated environment is a powerful feature of Airflow that gives DAG authors full control over how their tasks run.

This type of pattern is frequently used when you:

  • Have one task that requires a different set of Python dependencies.
  • Have a task that relies on a Python version that’s different from the version used in your Airflow environment.
  • Have tasks that require specific resources that you don’t want to apply to every task.

Running tasks in a separate environment can help you avoid common data pipeline issues, like dependency conflicts or out-of-memory errors, and it can save resources. Airflow DAG authors have multiple options for running tasks in isolated environments. In this webinar, we’ll cover everything you need to know, including:

  • How to use the KubernetesPodOperator, PythonVirtualEnvOperator, and ExternalPythonOperator.
  • How to choose between these operators based on your use case.
  • How to use decorators to implement these operators for clean and easy-to-read DAG code.
  • How to set up your Airflow infrastructure to support running tasks in isolated environments.

Tamara Fingerlin - Developer Advocate at Astronomer

Tamara is a Developer Advocate at Astronomer, the commercial developer behind Apache Airflow. She discovered her passion for data and optimizing workflows in her previous career and now focuses on creating educational resources that help beginners to get started with Airflow. Being a career changer herself, she cares deeply about making exciting Airflow features accessible for users of all backgrounds. In her spare time she helps to organize data challenges as a Data Scientist at the World Data League, a competition tackling social impact challenges.

Kenten Danas - Lead Developer Advocate at Astronomer

Kenten is a Lead Developer Advocate at Astronomer, with a background in field engineering, data engineering, and consulting. She has first-hand experience adopting and running Airflow as a consultant, and is passionate about helping other data engineers scale and get the most out of their Airflow experience.

Astronomer Webinars are biweekly, real-time online sessions for data pipeline authors hosted by Astronomer’s Apache Airflow experts. During an hour-long meeting, participants have a chance to dive into the most important features and practices related to Apache Airflow and data orchestration — from Airflow 2+ feature highlights to DAG writing best practices. At the end of each webinar, we open the floor for a Q&A to ensure that participants leave the event confident about their newly acquired knowledge.

Hosted By

  • Tamara Fingerlin Tamara Fingerlin Developer Advocate
    Astronomer
  • Kenten Danas Kenten Danas Lead Developer Advocate
    Astronomer