> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Hybrid Search for eCommerce reference architecture

The Hybrid Search for eCommerce [GitHub repository](https://github.com/astronomer/astronomer-weaviate-hybrid-search) is a free and open-source reference architecture showing how to use [Apache Airflow®](https://airflow.apache.org/) with [Weaviate](https://weaviate.io/) to build an automated hybrid search application. A demo of the architecture was shown in the [Modern Infrastructure for World Class AI Applications](https://www.astronomer.io/events/webinars/modern-infrastructure-for-world-class-ai-applications-video/) webinar.

<Frame>
  <img src="https://mintcdn.com/astronomer/OKshYsmXSJEKC_h3/images/img/reference_architectures/reference-architecture-hybrid-search_web_interface.png?fit=max&auto=format&n=OKshYsmXSJEKC_h3&q=85&s=3395d06ab8dfa0a0e69af3536187129e" alt="Screenshot of the Hybrid Search application frontend." width="1891" height="941" data-path="images/img/reference_architectures/reference-architecture-hybrid-search_web_interface.png" />
</Frame>

This reference architecture demonstrates how to use Apache Airflow to orchestrate RAG data ingestion that powers a search application as well as a batch inference pipeline analyzing search queries. It also shows how to use Weaviate's advanced search capabilities. You can adapt the Hybrid Search application to your use case by ingesting your own data and adjusting the search queries in the website backend to fit your needs.

## Architecture

<Frame>
  <img src="https://mintcdn.com/astronomer/OKshYsmXSJEKC_h3/images/img/reference_architectures/reference-architecture-hybrid-search_architecture_diagram.png?fit=max&auto=format&n=OKshYsmXSJEKC_h3&q=85&s=3926fc00498dd3f3394e7cc8dfdc591b" alt="Hybrid search reference architecture diagram." width="13743" height="7344" data-path="images/img/reference_architectures/reference-architecture-hybrid-search_architecture_diagram.png" />
</Frame>

The hybrid search reference architecture consists of 3 main components:

* **Data ingestion and embedding**: Sample data containing product descriptions and images is ingested from [Amazon S3](https://aws.amazon.com/s3/) and [Snowflake](https://www.snowflake.com/) into [Weaviate](https://weaviate.io/), a vector database. Embedding of the product descriptions uses [OpenAI](https://platform.openai.com/docs/api-reference/introduction) models.
* **Hybrid search**: The demo website with a [Flask](https://flask.palletsprojects.com/en/2.0.x/) backend and [React](https://reactjs.org/) frontend allows users to experiment with advanced Weaviate search by querying the product descriptions using hybrid search. An OpenAI embedding model is used to embed the user query.
* **Batch inference**: All user search queries are stored back in Weaviate so they can be used by a downstream Airflow DAG that runs an OpenAI batch inference pipeline to classify user queries and derive product insights. The results of this analysis are loaded into Snowflake to be displayed in a [Streamlit](https://streamlit.io/) dashboard.

### Airflow features

The DAGs that power this hybrid search application highlight several key Airflow best practices and features:

* [Airflow retries](/docs/learn/2.x/rerunning-dags#automatically-retry-tasks): To protect against transient API failures and rate limits, all tasks are configured to automatically retry after an adjustable delay.
* [Advanced data-driven scheduling](/docs/learn/2.x/airflow-datasets): The DAGs in this reference architecture run on data-driven schedules, including combined [dataset and time scheduling](/docs/learn/2.x/airflow-datasets#combined-dataset-and-time-based-scheduling) and [conditional dataset scheduling](/docs/learn/2.x/airflow-datasets#conditional-dataset-scheduling).
* [Dynamic task mapping](/docs/learn/2.x/dynamic-tasks): Product information extraction and ingestion into Weaviate are split into multiple parallelized tasks, the number of which is determined at runtime based on the number of ingestion folders with product information that needs to be processed.
* [Object Storage](/docs/learn/2.x/airflow-object-storage-tutorial): Interaction with files in object storage is simplified using the experimental Airflow Object Storage API.
* Modularization: Functions defining how information is extracted and checksums are calculated are modularized in the [`include`](https://github.com/astronomer/astronomer-weaviate-hybrid-search/blob/main/include/functions/utils.py) folder and imported into the DAGs. This makes the DAG code more readable and offers the ability to reuse functions across multiple DAGs.

## Next steps

Get the [Astronomer GenAI cookbook](https://www.astronomer.io/ebooks/gen-ai-airflow-cookbook/?utm_source=learn-docs-reference-architectures\&utm_medium=web\&utm_campaign=gen-ai) to view more examples of how to use Airflow to build generative AI applications.

If you'd like to build your own hybrid search application, feel free to fork the [repository](https://github.com/astronomer/astronomer-weaviate-hybrid-search) and adapt it to your use case. We recommend to deploy the Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/?utm_source=learn-docs-reference-architectures\&utm_medium=web\&utm_campaign=free-trial).