> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# SnowPatrol: Snowflake Usage Anomaly Detection & Alerting System
This is a reference architecture meant to serve as inspiration for how to use Airflow. The SnowPatrol repository may not be actively maintained. If you're looking for production-ready Snowflake cost management, consider [Astro Observe](https://www.astronomer.io/product/observe/).
[SnowPatrol](https://github.com/astronomer/snowpatrol) is an anomaly detection and alerting system for [Snowflake](https://www.snowflake.com/) built with [Apache Airflow®](https://airflow.apache.org/). It uses an [Isolation Forest](https://en.wikipedia.org/wiki/Isolation_forest) model to detect anomalies in your Snowflake compute driving cost. Airflow Dags orchestrate feature engineering, model training, and prediction. Anomaly records are stored in a Snowflake table and alerts are sent to a [Slack](https://slack.com/) channel. The full source code is open-source and available on [GitHub](https://github.com/astronomer/snowpatrol).
SnowPatrol serves a dual purpose:
* **Resource for Airflow + Snowflake users**: Many organizations use Snowflake and have experienced challenges in managing associated costs, especially incurred from virtual warehouse compute. SnowPatrol is a tool organizations can use to be alerted of anomalies in their Snowflake compute usage, and take action to reduce costs.
* **Learning Tool**: SnowPatrol is an MLOps reference implementation, showing how you can use Airflow to train, test, deploy, and monitor predictive models. The structure of the Dags can be adapted to other use cases, such as fraud detection.
## Architecture
SnowPatrol performs the following steps:
* **Data Ingestion and Feature Engineering**: Snowflake usage data is persisted in a Snowflake table to create a cost time series. [STL decomposition](https://www.statsmodels.org/dev/examples/notebooks/generated/stl_decomposition.html) is used to extract trends, seasonality, and residuals.
* **Model Training**: An Isolation Forest model is trained on the features to detect anomalies. Model training and versions are tracked using [Weights & Biases](https://wandb.ai/site).
* **Predictions**: The trained model is used to predict anomalies in Snowflake usage data.
* **Reporting and Alerting**: Anomalies are stored in a Snowflake table and alerts are sent to a Slack channel.
### Airflow features
The Dags that power SnowPatrol highlight several key Airflow best practices and features:
* [Dynamic task mapping](/docs/learn/dynamic-tasks): Every time the Dags run, the list of warehouses in Snowflake is determined at runtime and Airflow automatically creates a dynamically mapped task instance per warehouse for model training and prediction to run in parallel.
* [Data-aware scheduling](/docs/learn/airflow-datasets): While the first Dag runs on a time-based `@daily` schedule, all other Dags are triggered based on updates to the datasets they depend upon.
* [Notifications](/docs/learn/error-notifications-in-airflow): If any task fails, a Slack notification is automatically sent using an `on_failure_callback` in the `default_args` of the Dags.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): All tasks are configured to automatically retry after a set delay.
* Modularization: SQL statements are stored in the [`include`](https://github.com/astronomer/snowpatrol/tree/main/include/sql) folder and executed by [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) in the Dag. This makes the Dag code more readable and offers the ability to reuse SQL queries across multiple Dags.