WEBINAR

Running Airflow Tasks in Isolated Environments

Recorded On November 29, 2022

  • Tamara Fingerlin
  • Kenten Danas

This webinar provides an extensive overview of the different options for running Airflow tasks in isolated environments - a Kubernetes pod or Python virtual environment that’s separate from your Airflow environment. The ability to run tasks in an isolated environment helps avoid common data pipeline problems, like dependency conflicts and resource management, and gives DAG authors control over how and where their tasks run. It’s useful, for example, if you need to run a task requiring Python packages that conflict with core Airflow.

Questions covered in the webinar include:

  • What are the use cases for running Airflow tasks in isolated environments?
  • What operators should I use to run tasks in an isolated environment?
  • How do I use the suggested operators?
  • How do I set up my Airflow infrastructure to support running tasks in isolated environments?

All of the sample code shown in this webinar can be found in this repo.

See More Resources

Dynamic Task Mapping on Multiple Parameters

The Airflow Templates VS Code Extension

Efficient data quality checks with Airflow 2.7

How to Orchestrate Machine Learning Workflows with Airflow

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.