Astronomer Providers

Watch On Demand

Hosted By

  • Kenten Danas
  • Phani Kumar

1. What are Astronomer Providers?

Astronomer Providers are a new set of open source providers created and maintained by Astronomer to support workloads and use cases that benefit from running asynchronously. These Apache 2-licensed providers are built for compatibility with OSS Airflow and will be supported and maintained long-term by Astronomer. All Astronomer Providers will soon be available on the Astronomer registry.

Our latest release includes 35 operators and sensors. Documentation can be found here.

2. What is an Async Operator?


An async operator (also known as a deferrable operator) is an operator or a sensor that creates efficiencies in the utilization of Airflow worker slots. Unlike normal operators and sensors, which take up a full worker slot for the entire time they are running (even if they are idle), async operators have the ability to vacate the worker slot and free the worker to complete other tasks.

Example: If you have 100 worker slots available to run tasks and you have 100 DAGs waiting on a Sensor that’s currently running, but idle, you will not be able to run anything else — even though your entire Airflow cluster is idle. Async operators help solve this problem by deferring the task in the worker slot, making it available for other tasks. The async operator is also equipped with a callback function that resumes the deferred task. This process of deferral and resumption is managed by the triggerer, an Airflow component run separately from the scheduler; more detail on what this process can and cannot do is explained in the video.

Below is an example structure of an async operator:


3. Why Use Async Operators and Sensors?


4. When Should You Make an Operator or Sensor Async?

You can create async operators for the “sync-version” of operators that take more than a few minutes to complete.

For example, you won’t create an async operator for a BigQueryCreateEmptyTableOperator, which should run quite quickly, but you will create one for BigQueryInsertJobOperator, which actually runs queries and can take hours (in the worst-case scenario) for task completion.

Some example use cases:

5. Best Practices

When considering using Astronomer Providers, we recommend taking the following best practices into consideration.