The past few years have brought a dramatic shift in the data landscape—as data gets bigger and faster, many organizations find themselves spending more time centralizing and preparing data sets for analytics than generating insights. In order to combat this, companies have invested heavily in extract, transform, load (or ETL) tools, but old-school drag-and-drop tools no longer cut it for savvy data teams who need a more flexible solution.
To satisfy this need for more customizability, AirBnb Data Engineer Maxime Beauchemin created and then open-sourced Airflow: a workflow management system that defines tasks and their dependencies as code and then distributes and executes them on a defined schedule. Built by developers, for developers, Airflow is based on the principle that ETL is best expressed in code.
Why We Created Astronomer
While Airflow is ambitious in design and vision, it can cause some headaches, particularly with respect to implementation, devops, and maintenance. Our team at Astronomer has built a Managed Airflow product to accomplish three major goals:
1. Secure deployment and scaling
Fast and secure deployment to a managed cloud environment and seamless horizontal scaling ensures that time is spent on writing data pipelinesinstead of dealing with infrastructure issues.
2. Provide a world-class developer experience
Astronomer caters to the needs and desires of modern open-source developers with lightweight tools, a rich CLI and API, and a locally mirrored dev environment.
3. Work with data anywhere
Proliferation of SaaS silos and internal straddling of cloud/on-premise data environments means that data teams must be prepared to work with a variety of data in the cloud across technical and corporate barriers.
Get Started With Apache Airflow
If you’re interested in running Apache Airflow on Astronomer, reach out to us today. For guidelines on how to install Airflow on your local machine with Astronomer’s open source CLI, check out our guides.
