We’re proud to announce the latest version of our commercially-supported, high-availability solution for organizations seeking to adopt Apache Airflow.
We really like Airflow
As we have mentioned a couple of times before, we are all-in on Airflow–and for good reason. Airflow is sweeping the data landscape, with huge companies like Pandora and RobinHood integrating the scheduling and orchestration tool into their data infrastructure. Through the many conversations we've had with folks in the Airflow community, we've found that the biggest pain point for adoption is that setting up the infrastructure to support, secure, and scale a diverse and reliable ecosystem of Airflow clusters is expensive and requires a huge team of nerds. That's the problem we are solving here at Astronomer.
With that, we’d like to formally introduce you to the latest version of our managed Airflow product: Astronomer v0.3. It's jam-packed full of new features and optimizations. Let's dive in.
Meet Astronomer v0.3
Our team has continued to build on our last release, which laid the foundation for easy Apache Airflow deployments in a user’s own, secure environment.
Astronomer v0.3 brings improvements across the board, including optimizations to our CLI, authentication methods, monitoring stack, out-of-the-box security, scalability with Kubernetes, and a brand new UI for simplified deployment of multiple Airflow clusters with custom permission groups.
The latest release allows users to spin up multiple Airflow clusters anywhere Kubernetes runs, including both public clouds (AWS, GCP, Azure) and on-prem private clouds. It is designed to simplify the process of running and monitoring data workflows and clusters at scale, with high-availability and maximum security. Now, you can access and monitor Celery worker queues right in the new UI–and that's just the tip of the iceberg regarding our plans to improve Airflow monitoring.
While this release allows our infrustructure to scale up to support larger organizations with many workflows and complex requirements, it also allows us to scale down to serve smaller teams (we can even run on a single-node Kubernetes cluster!). Astronomer v0.3 is designed to greatly reduce the time and effort required to get Airflow running in a stable and secure way, even at scale.
“Astronomer provides a turn-key, flexible, scalable, and affordable ETL solution to power our batch processing of billions of rows of data per day for our customers. In addition, their expert data engineers have trained our engineering team so we could be self-sufficient; they've been a great partner! - Nic Zangre, VP Product, CaliberMind
Our Newest Release, Unpacked
Here are a few specific benefits and features of the v0.3 release:
Multiple clouds and teams.
Once Astronomer is installed, users are up and running with the ability to deploy multiple Airflow clusters. Users can create team workspaces that are isolated, with the ability to invite team members to collaborate.
Brand new UI.
We made it incredibly easy to create and manage these Airflow clusters and teams with just a few clicks in our new UI, as well as some included monitoring of teams and Celery workers. The future is bright for this tool, as it will soon be home-base for your resource montoring and controls, as well as Airflow logs and alerts.
Astronomer relies exclusively on Kubernetes to provide maximum uptime and reliability, and this version brings even deeper integration. We have seen big improvements in availability with v0.3 installs vs competitors such as Google's Cloud Composer, as well as previous versions of our own product.
We drastically improved our authentication process for maximum security and usability, and made several additional improvements to our configuration process from start to finish.
Try it Out
Along with its platform subscriptions that include Airflow user support, Astronomer also offers Airflow professional services and training, as well as a growing library of guides, podcast episodes and blog posts. Recently, Astronomer also launched Data Engineering meetups in Denver, San Francisco and Cincinnati.