Build Event-Driven Workflows with Apache Airflow®: On-Demand Webinar
Watch the on-demand webinar on building and operating event-driven pipelines with the latest Airflow features.
Assumed knowledge
To get the most out of this guide, you should have an existing knowledge of:- Airflow assets. See Assets and data-aware scheduling in Airflow.
Concepts
There are a number of concepts that are important to understand when using event-driven scheduling.- Data-aware scheduling: Data-aware or data-driven scheduling refers to all ways you can schedule a DAG based on updates to assets. Updates to assets outside of event-driven scheduling occur by tasks in the same Airflow instance completing successfully, manually through the Airflow UI, or through a call to the Airflow REST API. See Assets and data-aware scheduling in Airflow.
- Event-driven scheduling: Event-driven scheduling is a sub-type of data-aware scheduling where a DAG is run based on messages posted to a message queue. This message in the queue is triggered by an event that occurs outside of Airflow.
- Message queue: A message queue is a service that allows you to send and receive messages between different systems. Examples of message queues include Amazon SQS, RabbitMQ, and Apache Kafka. Event-driven scheduling in Airflow 3.0 is supported for Amazon SQS with support for other message queues planned for future releases.
- Trigger: A trigger is an asynchronous Python function running in the Airflow triggerer component. Triggers that inherit from
BaseEventTriggercan be used in AssetWatchers for event-driven scheduling. The trigger is responsible for polling the message queue for new messages, when a new message is found, aTriggerEventis created. The message is deleted from the queue. - AssetWatcher: An AssetWatcher is a class in Airflow that watches one or more triggers for events. When a trigger fires a
TriggerEvent, the AssetWatcher updates the asset it is associated with, creating anAssetEvent. The payload of the trigger is attached to theAssetEventin itsextradictionary. - AssetEvent: An
Assetis an object in Airflow that represents a concrete or abstract data entity. For example, an asset can be a file, table in a database, or not tied to any specific data. AnAssetEventrepresents one update to an asset. In the context of event-driven scheduling, theAssetEventrepresents one message having been detected in the message queue.
In Airflow versions before 3.4, asset events that occurred in quick succession were sometimes batched into one Dag run. In Airflow 3.4 and later, each
AssetEvent creates its own Dag run by default, so a Dag scheduled on an asset with an AssetWatcher runs once per AssetEvent. To explicitly control batching behavior, see Batch asset events.When to use event-driven scheduling
Basic and advanced data-aware scheduling are great for use cases where updates to assets occur within Airflow or can be accomplished through a call to the Airflow REST API. However, there are scenarios where you need a DAG to run based on events in external systems. Two common patterns exist for event-driven scheduling:- Data delivery to an external system: Data is delivered to an external system, such as manually by a domain expert, and a data-ready event is sent to a message queue. The DAG in Airflow is scheduled based on this message event and runs an extract-transform-load (ETL) pipeline that processes the data in the external system.
- IoT sensor events: An Internet of Things (IoT) device sends a sensor event to a message queue. A DAG in Airflow is scheduled based on this message event and consumes the message to evaluate the sensor value. If the evaluation determines that an alert is warranted, an alert event is published to another message queue.

None provided as the logical_date, meaning simultaneous triggering of multiple DAG runs is possible.
Example: Amazon SQS
This example shows how to configure a DAG to run as soon as a message is posted to an Amazon SQS queue.- Create an Amazon SQS queue. See Amazon Simple Queue Service Documentation for instructions.
-
Add the Airflow Common Messaging provider and Airflow Amazon provider to your Airflow instance. When using the Astro CLI, you can add the providers to your
requirements.txtfile: -
Set the connection to your Amazon SQS queue in your Airflow instance. Note that the connection needs to include the
region_namein theextrafield. Replace<your access key>,<your secret key>, and<your region>with your AWS credentials and region. For other authentication options see the Airflow Amazon provider documentation.Your AWS user requires at leastsqs:ReceiveMessageandsqs:DeleteMessagepermissions. -
Create a new file in the
dagsfolder of your Airflow project and add the following code. Replace theSQS_QUEUEwith the URL to your message queue.This DAG is scheduled to run as soon as thesqs_queue_assetasset is updated. This asset uses oneAssetWatcherwith the namesqs_watcherthat watches oneMessageQueueTrigger. This trigger is polling for new messages in the provided SQS queue. Theprocess_messagetask gets the triggering asset events from the Airflow context and prints the message body from the triggering message. Theprocess_messagetask is a placeholder for your own task that processes the message. -
Create a new message in the SQS queue. It will trigger the DAG to run and the
process_messagetask will print the message body.
Example: Apache Kafka
To use Apache Kafka as the message queue for event-driven scheduling, follow these steps:- Create an Apache Kafka topic. See the Apache Kafka documentation for instructions.
-
Add the Airflow Common Messaging provider and Airflow Apache Kafka provider to your Airflow instance. When using the Astro CLI, you can add the providers to your
requirements.txtfile: -
Set the connection to your Apache Kafka topic in your Airflow instance. Below is an example connection JSON, depending on your Kafka setup you may need to provide additional values in the
extradictionary. -
Create a new file in the
includefolder of your Airflow project calledkafka_trigger.pyand add the following code. This function will be applied to the Kafka message as soon as it is received and return the value of the message to be processed by the Dag. -
Create a new file in the
dagsfolder of your Airflow project and add the following code. Replace the placeholders in theKAFKA_QUEUEwith your information.This DAG is scheduled to run as soon as thekafka_topic_assetasset is updated. This asset uses oneAssetWatcherwith the namekafka_watcherthat watches oneMessageQueueTrigger. This trigger is polling for new messages in the provided Kafka topic. Theprocess_messagetask gets the triggering asset events from the Airflow context and prints the event information from the trigger event that includes the message. Theprocess_messagetask is a placeholder for your own task that processes the message. -
Create a new message in the Kafka topic. It will trigger the DAG to run and the
process_messagetask will print the event information.
Write a custom event trigger
If no existing trigger supports your event source, you can write your own by subclassingBaseEventTrigger. A custom event trigger needs to:
- Implement
serialize(), which returns the import path of the trigger class and the keyword arguments needed to re-create it. The triggerer calls this method to store the trigger in the metadata database. - Implement
run(), an async generator that yields aTriggerEventwhen it detects an event. Airflow turns eachTriggerEventinto anAssetEvent, and the payload becomes theextraof that event.
self.asset_state_store before it calls run(). This attribute is an accessor for the asset state store of the asset the trigger watches. Use it to persist a watermark, such as the timestamp of the last record the trigger processed, so the trigger doesn’t fire again for the same data each time it runs.
self.asset_state_store is None in __init__ and serialize(), and is only available inside run(). Plain BaseTrigger subclasses, which are used to defer tasks, don’t have access to the asset state store.Example: Watch a SQL table for new records
The following trigger polls a SQL table for rows with a timestamp later than a stored watermark. When it finds new rows, it fires oneTriggerEvent per row and moves the watermark to the latest timestamp it saw.
Prerequisites
The following is required for this example:- Airflow 3.3 or later.
- The Airflow Common SQL provider and the provider for your database installed in your Airflow environment.
- An Airflow connection to your database, with a table that has a timestamp column that increases as new records arrive.
Define the trigger
Create a file in theinclude folder of your Airflow project called sql_triggers.py and add the following code:
serialize()returns only the arguments needed to rebuild the trigger. It doesn’t include state such as the watermark, which lives in the asset state store.run()reads thewatermarkkey withaget. On the first poll the key doesn’t exist, so the trigger selects every row in the table.- The query runs in a worker thread with
asyncio.to_threadbecause the hook is synchronous. All triggers on a triggerer share one event loop, so blocking calls delay every other trigger. - After the trigger yields events, it stores the latest timestamp with
asetand returns. After the asset event is created, the triggerer starts the trigger again, and the next poll begins at the new watermark. run()uses the async methodsagetandaset. The synchronousgetandsetmethods also work, but they block the event loop for the full round trip to the API server.
Use the trigger in a Dag
Create an asset with anAssetWatcher that uses the trigger, then schedule a Dag on the asset:
process_order task reads the row from the payload key of the AssetEvent extra.
To read or reset the watermark, open the asset in the Airflow UI and go to the Asset State Store tab. Deleting the watermark entry makes the trigger process every row in the table again on its next poll.