Bringing Big Data to Small Businesses
Anastasia addresses the pressing challenges faced by small and medium-sized enterprises (SMBs) in optimizing their operations, predicting sales, and managing inventory. In a business landscape where precision, efficiency, and customer retention are paramount, many SMBs grapple with the complexities of data-driven decision-making.
Anastasia’s solution emerges as a solution, harnessing the power of artificial intelligence (AI) to provide actionable insights, streamline inventory management, and predict sales trends. By making AI accessible and user-friendly for these businesses, Anastasia not only enhances their operational efficiency but also empowers them to focus on core business activities, ensuring they remain competitive and profitable in a dynamic market.
Anastasia’s platform is built on foundational principles that prioritize the needs of SMBs. Emphasizing simplicity, they ensure that businesses receive clear and actionable insights without the complexities often associated with AI systems. Their unique engineering approach involves proprietary generative AI that seamlessly integrates various AI modules to deliver precise answers. The platform’s automation capabilities allow for effortless integration with existing e-commerce and ERP systems, streamlining data collection and analysis.
Furthermore, their commitment to efficiency is evident in their AI’s ability to deliver reliable results using significantly less computing power than competitors. Anastasia stands out with its one-click functionality, rapidly structuring sales and inventory data into human-readable formats without end-user intervention. Within a mere 24 hours after querying, businesses are equipped with answers to pivotal questions, from inventory decisions to sales predictions, ensuring they’re always a step ahead in their market.
From AWS Step Functions to Fully-Hosted Airflow
When Anastasia was in its nascent stages, the platform primarily targeted larger companies and focused on broad predictions rather than the refined “what to buy” insights it offers today. At that time, they utilized AWS Step Functions to orchestrate their AI workloads, encompassing preprocessing, inference, and post-processing in the machine learning life cycle.
This approach involved orchestrating around four or five processes. However, as the platform evolved to cater to smaller businesses that required more actionable data, challenges with step functions became evident. The step functions, while efficient, operated within a confined environment, often likened to a “walled garden.” This limited flexibility and integration capabilities.
Moreover, while tools like AWS Lambda could be invoked within these functions, provisioning the necessary infrastructure for scalability wasn’t straightforward. The team sought a more robust approach, leveraging tools like Terraform for infrastructure as code (IaC) and managing different environments. This led to their transition to a hosted Airflow on Amazon Elastic Container Service (ECS). One of the primary reasons for this shift was Airflow’s scalable compute infrastructure and the ease of integrating Python logic, offering a more streamlined and scalable solution compared to the constraints of step functions.
Anastasia’s journey with Airflow began with a self-managed setup, which served them well for a significant period, experiencing minimal downtime. However, as the platform grew and evolved, the team found themselves grappling with the challenges of maintaining Airflow, especially when considering upgrades. They were using Airflow version 1.x, and the prospect of migrating to newer versions seemed daunting due to time constraints and other pressing priorities. This is when they came across Astronomer’s offering for a hosted Airflow solution.
“It Just Clicks” with Astronomer
“The appeal of Astronomer was multifaceted. It not only promised an easier migration and maintenance process but was also up-to-date with the latest Airflow versions,” explained Juan Honorato, Chief Technology Officer, Anastasia. “Our primary reason for moving to Astronomer to manage Airflow was that we didn’t have the resources or enough time to maintain Airflow.”
While they did evaluate other services, such as Amazon Managed Workflows for Apache Airflow (MWAA), they found it lacking in terms of updates and the overall developer experience. In contrast, Astronomer’s developer experience stood out as top-notch. “The intuitive nature of Astro and the immediate positive feedback from the “Astro dev start” command showcased Astronomer’s commitment to a seamless user experience. As soon as you try the Astro CLI, you notice right away that the developer experience is first class,” said Honorato.
This “it just clicks” sentiment, as expressed by Honorato, combined with the evident dedication of the Astronomer team, solidified Anastasia’s decision to transition to their hosted Airflow solution.
Upon Anastasia’s decision to transition to Astronomer, they were faced with the challenge of migrating from Airflow version 1.x to Astronomer’s 2.x. This migration was not just about transitioning to a new platform but also about upgrading to a more advanced version of Airflow.
One of the compelling factors in favor of Astronomer was its significant contribution to Airflow, with an estimated 70-80% of pull requests in Airflow originating from Astronomer. This meant that while AWS offered no assistance in the migration, Astronomer was well-equipped to guide Anastasia through the process. When it came to rewriting their DAGs, the process was relatively straightforward. Only a few operators needed updates, and with Astronomer’s guidance, Anastasia learned best practices in DAG creation and management.
“The support from Astronomer was hands-on, with multiple paired programming sessions ensuring a smooth migration and that we were leveraging the full potential of the latest Airflow version. The entire experience underscored the value of partnering with a dedicated and knowledgeable team, ensuring that the migration and upgrade were both efficient and optimized,” praised Honorato.
Unlocking Possibilities and Embracing the Future
Upon transitioning to Astronomer, Anastasia experienced a newfound confidence in their Airflow operations. “The ease of configuration adjustments on Astronomer meant that even if there were potential missteps or heavy reliance on workers, the team could swiftly react and rectify. This confidence in our operations led to a shift in how we’re using Airflow today and allowing us to explore new cases,” said Honorato.
Previously, their primary use was orchestrating services, often resorting to calling ECS tasks even for simple API calls. With Astronomer, they began embedding more Python logic directly into their workflows, leading to enhanced flexibility and a noticeable boost in development speed.
One significant advancement was their collaboration with AnyScale, known for the Ray library focused on distributed computing for machine learning. Anastasia developed their own AnyScale operators, leveraging deferrable operators, to manage their ML workloads. This was a departure from their earlier hesitance to experiment with custom operators due to potential maintenance challenges for their compact team.
Another transformative change was their shift from a single-job-per-client model to a multi-tenant approach in Airflow. This was in line with their introduction of a premium model and partnerships with data providers, which saw them transitioning from managing around 100 clients to over 15,000. This scalability and flexibility underscored Airflow’s adaptability to their evolving needs, further solidifying its integral role in Anastasia’s operations.
Anastasia is poised for an exciting future with Airflow and Astronomer, with several initiatives on the horizon that promise to further enhance their operations. One of the key developments they’re enthusiastic about is using Cosmos to integrate dbt (Data Build Tool) into their workflows. Additionally, Anastasia is keen on transitioning to a more event-driven architecture. They’ve already made strides in this direction with an integration from EventBridge that triggers Airflow, a capability enhanced by the robust API offerings of Airflow 2.
This shift towards event-based messaging is still a work in progress, and while the exact path is yet to be finalized, the potential benefits are evident. Another significant enhancement that has added value to their operations is the move to a parameter store backend, a recommendation from the Astronomer onboarding process. This change has streamlined their configuration management, reducing manual interventions and ensuring better tracking of different versions.
The parameter store’s utility has been so impactful that Anastasia is now extending its reach to other processes, APIs, and even facilitating intercommunication between different workspaces. This evolution, coupled with the continuous support and insights from Astronomer, underscores Anastasia’s commitment to leveraging the best tools and practices to drive their business forward.
Anastasia is a cutting-edge technology company specializing in providing AI-driven solutions to small and medium-sized enterprises. Their service empowers SMBs to optimize operations, predict sales, and manage inventory efficiently, enabling them to thrive in a competitive business landscape.
Initially, Anastasia targeted larger companies and had to evolve to cater to the specific needs of SMBs, who required more actionable and user-friendly data insights. They faced difficulties with AWS step functions, which limited flexibility and scalability in orchestrating their AI workloads. Additionally, they needed to upgrade from Airflow version 1.x to the more advanced Airflow 2.x to keep up with the evolving technology landscape.
The Astro Solution
Astronomer provided Anastasia with a hosted Airflow solution, Astro, to overcome their challenges. Astro offered a seamless upgrade to Airflow 2.x and simplified the migration process. Astronomer's developer tools, such as the Astro CLI, ensured a user-friendly experience for Anastasia's team during the transition.
Anastasia's transition to Astronomer's Airflow solution, including the latest Airflow versions, significantly enhanced scalability. This allowed them to effectively meet the needs of a growing number of SMB clients.
Anastasia's team benefited from no longer having to manage Airflow themselves, as well as confidence in their environment's integrity. This gave them the time and peace of mind to explore new use cases for Airflow.
Improved Development Speed
By collaborating with the Astronomer team to solve issues and learn best practices, Anastasia was able to do more with Airflow, faster.
Enhanced ML Workloads
Anastasia successfully managed machine learning workloads with the introduction of custom AnyScale operators, further advancing their capabilities.
Shifting to a multi-tenant approach in Airflow allowed Anastasia to efficiently manage a larger client base, supporting their transition from managing 100 clients to over 15,000.
Anastasia is now well-prepared for the future with plans to integrate DBT into their workflows, transition to an event-driven architecture, and optimize data infrastructure for scalability, ensuring continued growth and success.