Create Powerful Data Pipelines by Mastering Sensors

WATCH ON DEMAND

Summary:

Do you use Sensors in your data pipelines? Do you need to wait for a file before executing the next step? Are you looking to execute your task after a task completes in another DAG? Would you like to wait for an import in your SQL table before executing the next task? The answer? Airflow Sensors! Sensors are pretty basic in their usage. You wait for a condition to be true. It’s as simple as that. But... That’s not all! Not only are there some arguments which allow you to make your sensors more reliable, but there are also a large number of sensors covering a wide range of use cases. By the end of this webinar, from basic to advanced, Sensors won’t hold any more secrets for you.

A few key takeaways:

  • Configure your Sensors to avoid freezing your Airflow instance
  • Dive into the ExternalTaskSensor
  • Smart Sensors, what are they? Should you use them?

Missed the Webinar? Sign up for the Recap

Recap Preview

Webinar agenda:

  1. What is a sensor? A reminder
  2. Use cases
  3. Sensor types
  4. Implementing a sensor + best practices
  5. Smart sensors

1. What is a sensor?

master sensor 1

A sensor is a kind of operator waiting for something to happen before moving to the next task. How a sensor works:

  1. Checks if a given condition is true
  2. If it is true - then the next task is being executed
  3. Otherwise - it will wait for a time interval you have defined and check the condition again

It’s crucial if, for example, you’re waiting for a file or SQL entry.

Hosted By

Marc Lamberti

Marc Lamberti

Head of Customer Training

Viraj Parekh

Viraj Parekh

Field CTO