How Anastasia Delivers AI-Powered Insights with Astro

Bringing Big Data to Small Businesses

Anastasia addresses the pressing challenges faced by small and medium-sized enterprises (SMBs) in optimizing their operations, predicting sales, and managing inventory. In a business landscape where precision, efficiency, and customer retention are paramount, many SMBs grapple with the complexities of data-driven decision-making. 

Anastasia's solution emerges as a solution, harnessing the power of artificial intelligence (AI) to provide actionable insights, streamline inventory management, and predict sales trends. By making AI accessible and user-friendly for these businesses, Anastasia not only enhances their operational efficiency but also empowers them to focus on core business activities, ensuring they remain competitive and profitable in a dynamic market.

Anastasia's platform is built on foundational principles that prioritize the needs of SMBs. Emphasizing simplicity, they ensure that businesses receive clear and actionable insights without the complexities often associated with AI systems. Their unique engineering approach involves proprietary generative AI that seamlessly integrates various AI modules to deliver precise answers. The platform's automation capabilities allow for effortless integration with existing e-commerce and ERP systems, streamlining data collection and analysis. 

Furthermore, their commitment to efficiency is evident in their AI's ability to deliver reliable results using significantly less computing power than competitors. Anastasia stands out with its one-click functionality, rapidly structuring sales and inventory data into human-readable formats without end-user intervention. Within a mere 24 hours after querying, businesses are equipped with answers to pivotal questions, from inventory decisions to sales predictions, ensuring they're always a step ahead in their market.

From AWS Step Functions to Fully-Hosted Airflow

When Anastasia was in its nascent stages, the platform primarily targeted larger companies and focused on broad predictions rather than the refined "what to buy" insights it offers today. At that time, they utilized AWS Step Functions to orchestrate their AI workloads, encompassing preprocessing, inference, and post-processing in the machine learning life cycle. 

This approach involved orchestrating around four or five processes. However, as the platform evolved to cater to smaller businesses that required more actionable data, challenges with step functions became evident. The step functions, while efficient, operated within a confined environment, often likened to a "walled garden." This limited flexibility and integration capabilities. 

Moreover, while tools like AWS Lambda could be invoked within these functions, provisioning the necessary infrastructure for scalability wasn't straightforward. The team sought a more robust approach, leveraging tools like Terraform for infrastructure as code (IaC) and managing different environments. This led to their transition to a hosted Airflow on Amazon Elastic Container Service (ECS). One of the primary reasons for this shift was Airflow's scalable compute infrastructure and the ease of integrating Python logic, offering a more streamlined and scalable solution compared to the constraints of step functions.

Anastasia's journey with Airflow began with a self-managed setup, which served them well for a significant period, experiencing minimal downtime. However, as the platform grew and evolved, the team found themselves grappling with the challenges of maintaining Airflow, especially when considering upgrades. They were using Airflow version 1.x, and the prospect of migrating to newer versions seemed daunting due to time constraints and other pressing priorities. This is when they came across Astronomer's offering for a hosted Airflow solution. 

“It Just Clicks” with Astronomer

“The appeal of Astronomer was multifaceted. It not only promised an easier migration and maintenance process but was also up-to-date with the latest Airflow versions,” explained Juan Honorato, Chief Technology Officer, Anastasia. “Our primary reason for moving to Astronomer to manage Airflow was that we didn’t have the resources or enough time to maintain Airflow.”

While they did evaluate other services, such as Amazon Managed Workflows for Apache Airflow® (MWAA), they found it lacking in terms of updates and the overall developer experience. In contrast, Astronomer's developer experience stood out as top-notch. “The intuitive nature of Astro and the immediate positive feedback from the "Astro dev start" command showcased Astronomer's commitment to a seamless user experience. As soon as you try the Astro CLI, you notice right away that the developer experience is first class,” said Honorato.

This "it just clicks" sentiment, as expressed by Honorato, combined with the evident dedication of the Astronomer team, solidified Anastasia's decision to transition to their hosted Airflow solution.

Upon Anastasia's decision to transition to Astronomer, they were faced with the challenge of migrating from Airflow version 1.x to Astronomer's 2.x. This migration was not just about transitioning to a new platform but also about upgrading to a more advanced version of Airflow. 

One of the compelling factors in favor of Astronomer was its significant contribution to Airflow, with an estimated 70-80% of pull requests in Airflow originating from Astronomer. This meant that while AWS offered no assistance in the migration, Astronomer was well-equipped to guide Anastasia through the process. When it came to rewriting their DAGs, the process was relatively straightforward. Only a few operators needed updates, and with Astronomer's guidance, Anastasia learned best practices in DAG creation and management. 

“The support from Astronomer was hands-on, with multiple paired programming sessions ensuring a smooth migration and that we were leveraging the full potential of the latest Airflow version. The entire experience underscored the value of partnering with a dedicated and knowledgeable team, ensuring that the migration and upgrade were both efficient and optimized,” praised Honorato.

Unlocking Possibilities and Embracing the Future

Upon transitioning to Astronomer, Anastasia experienced a newfound confidence in their Airflow operations. “The ease of configuration adjustments on Astronomer meant that even if there were potential missteps or heavy reliance on workers, the team could swiftly react and rectify. This confidence in our operations led to a shift in how we’re using Airflow today and allowing us to explore new cases,” said Honorato. 

Previously, their primary use was orchestrating services, often resorting to calling ECS tasks even for simple API calls. With Astronomer, they began embedding more Python logic directly into their workflows, leading to enhanced flexibility and a noticeable boost in development speed. 

One significant advancement was their collaboration with AnyScale, known for the Ray library focused on distributed computing for machine learning. Anastasia developed their own AnyScale operators, leveraging deferrable operators, to manage their ML workloads. This was a departure from their earlier hesitance to experiment with custom operators due to potential maintenance challenges for their compact team. 

Another transformative change was their shift from a single-job-per-client model to a multi-tenant approach in Airflow. This was in line with their introduction of a premium model and partnerships with data providers, which saw them transitioning from managing around 100 clients to over 15,000. This scalability and flexibility underscored Airflow's adaptability to their evolving needs, further solidifying its integral role in Anastasia's operations.

Anastasia is poised for an exciting future with Airflow and Astronomer, with several initiatives on the horizon that promise to further enhance their operations. One of the key developments they're enthusiastic about is using Cosmos to integrate dbt (Data Build Tool) into their workflows. Additionally, Anastasia is keen on transitioning to a more event-driven architecture. They've already made strides in this direction with an integration from EventBridge that triggers Airflow, a capability enhanced by the robust API offerings of Airflow 2. 

This shift towards event-based messaging is still a work in progress, and while the exact path is yet to be finalized, the potential benefits are evident. Another significant enhancement that has added value to their operations is the move to a parameter store backend, a recommendation from the Astronomer onboarding process. This change has streamlined their configuration management, reducing manual interventions and ensuring better tracking of different versions. 

The parameter store's utility has been so impactful that Anastasia is now extending its reach to other processes, APIs, and even facilitating intercommunication between different workspaces. This evolution, coupled with the continuous support and insights from Astronomer, underscores Anastasia's commitment to leveraging the best tools and practices to drive their business forward.