Astronomer Accelerates AI Workflows with Integrations for Top LLM Providers

Astro Platform elevates AI infrastructure with production-ready Airflow integrations to streamline machine learning operations.

November 28, 2023 - Astronomer, the leader in modern data orchestration, today announced a new set of Apache Airflow™ integrations to accelerate LLMOps (large language model operations) and support AI use cases. Modern, data-first organizations are now able to connect to the most widely-used LLM services and vector databases with integrations across the AI ecosystem, including OpenAI, Cohere, pgvector, Pinecone, OpenSearch, and Weaviate.

By enabling data-centric teams to more easily integrate data pipelines and data processing with machine learning (ML) workflows, organizations can streamline the development of operational AI. Astro provides critical data-driven orchestration for these leading vector databases and natural language processing (NLP) solutions, driving the MLOps and LLMOps strategies behind the latest generative AI applications.

DataOps is at the center of all ML operations and is driving forward generative AI and LLM production. As the de facto standard for DataOps, Airflow is the foundation for all data architectures and is already widely used in the construction of LLMs and by thousands of ML teams. With pluggable compute and thousands of integrations in the data science toolkit, Astro (the fully managed Airflow service from Astronomer) is the ideal environment for building and driving ML initiatives.

Supporting the entire AI lifecycle, from prototype to production, Astro provides “day two operations” that include monitoring, alerting, and end-to-end lineage, and guarantees enterprise-grade uptime to help prevent critical outages to AI operations. Astro also prioritizes collaboration between data and ML engineers, from traditional data pipelines and getting ML production-ready, to building AI applications on Airflow.

”Organizations today are already relying on Astro and Airflow to harness the data required to fuel LLMs and AI. With these new integrations, we are now helping organizations realize the full potential of AI and natural language processing, and optimize their machine learning workflows,” said Steven Hillion, SVP of Data and AI at Astronomer. “These integrations put Astro at the foundation of any AI strategy, to better process complex and distributed volumes of data with the open source and proprietary frameworks that drive the current generative AI ecosystem.”

These integrations further extend the benefits of Astro and Airflow to an organization’s AI strategy by:

• Improving data lineage: Especially in AI, where data comes from thousands of different sources and goes through multiple complex transformations, the need for visibility and observability of ML pipelines cannot be understated. As more integrations and complexity are brought into AI applications, it can be difficult to pinpoint and identify the source of predictions as well as where – and why – things go wrong. Astronomer is offering a single integrated environment for the development and execution of mixed ETL (extract, transform and load) and ML workflows; this includes crucial visibility into how models change and where data is coming from to build trustworthiness and transparency and provide a framework for compliance.
• Data availability: Data is more distributed than ever and better integration with the whole modern data stack ensures more reliable and consistent delivery of data across the AI ecosystem. Now, Astronomer’s platform helps users create resilient data pipelines to fuel reliable generative AI deployments in production environments.
• Flexibility and agility: In today’s ever-changing AI environment, organizations are required to adopt and adapt to more complex AI models and strategies. Astronomer continues to expand Astro’s integrations with leading AI tools to give enterprises the flexibility and freedom necessary to evolve their AI strategies to meet their business needs.

”The world of LLMs is moving fast, so it’s important that developers build on flexible platforms that can adapt,” said Bob van Luijt, CEO & Co-Founder at Weaviate. “Using Apache Airflow and Weaviate together provides a flexible, open source foundation for building and scaling AI applications.”

Astronomer is also making Ask Astro, its LLM-powered chatbot, available in the Apache Airflow Slack Channel, and sharing the source code as a reference implementation. Ask Astro leverages a wealth of Airflow knowledge from Astronomer-specific documents across Github, Stack Overflow, Slack, the Astronomer Registry, and more, making it immediately available as a starting point for developers who are looking to operationalize their applications.

Learn more about Astronomer’s AI and ML initiatives at: www.astronomer.io/solutions/ai-ml.

About Astronomer

Astronomer designed Astro, a modern data orchestration platform, powered by Apache Airflow™. Astro enables data teams to build, run, and grow their mission-critical data pipelines on a single platform for all of their data flows. Astro dramatically reduces costs, improves developer productivity, and reliably delivers your most data-centric use cases. For more information, visit www.astronomer.io.

Apache® and Apache Airflow™are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.