Autodesk Uses Astronomer's Airflow-Powered Orchestration to Support Its Cloud Transformation

Autodesk develops software for architecture, engineering, construction, and other industries. Their products include AutoCAD, one of the most popular design tools of all time. The company is working to enrich its products with cloud-collaborative capabilities while also building out industry-specific cloud platforms.

Autodesk’s legacy scheduling and workflow-management software, Apache Oozie, was not the best choice for its cloud transformation initiatives, so Autodesk partnered with Astronomer to replace Oozie with Astronomer’s Airflow-powered orchestration platform. In about 12 weeks, and with help from Astronomer Professional Services, Autodesk migrated its business-critical workflows from Oozie to Airflow, helping pave the way for the company’s cloud transformation.

”This was by far the best professional services experience I’ve ever had,” says Nick Wilson, Senior Manager of the Autodesk Platform team.

A Software Legacy Can’t Keep Up with Demand

Oozie, designed for scheduling workflows in Hadoop and Hive, had been a sound investment for Autodesk in 2012; a decade on, however, the Oozie project had all but come to a dead end.

Oozie was holding Autodesk back from growing and transforming its business. The company hoped to adopt new DataOps development patterns, anticipating that DataOps would enable engineers to rapidly develop new data products. And Autodesk needed its data scientists to develop new business use cases for machine learning (ML), taking advantage of powerful and versatile open-source ML tools and frameworks.

Oozie is a poor fit for these priorities: it doesn’t have pre-built connections for the cloud services Autodesk used to create its data warehouse. It also doesn’t interoperate with Autodesk’s preferred transformation software, let alone with popular ML tools and frameworks.

”We made a strategic decision to adopt certain cloud services that worked well for our business, but it would have required significant investment on our part to make those services work with Oozie,” Wilson says. “Given the status of Oozie as a project on life support, that’s something we weren’t willing to do.”

Oozie’s shared-access security model was a limitation, because Autodesk couldn’t easily give users access to data or resources based on their roles. Provisioning access for new and existing users was time-consuming and constituted a significant operational burden for Autodesk’s staff.

DAG failures had also become more common, impacting the ability of Autodesk’s product engineering teams to do their work, and causing Autodesk to look for an Oozie exit ramp.

The Power of Airflow, Minus the Pain of Managing It

Autodesk leadership decided to fully migrate from Oozie onto Astronomer’s Airflow-powered orchestration platform.

The decision to use Airflow and partner with Astronomer was straightforward, Wilson says. Autodesk needed to replace Oozie with a platform capable of orchestrating complex workflows designed to connect on-prem resources with cloud services. Airflow clearly fit that bill.

And Airflow’s ability to integrate with everything from legacy mainframe systems to cloud serverless computing platforms also put it on Autodesk’s shortlist. But Airflow’s trump card was its huge community of active committers.

”At the scale we’re dealing with, Airflow was an obvious solution, and the tool we would be investing in had to have either the long-term support of a very active community, or the backing of an established software vendor,” Wilson says. “With the size of the Airflow community, we have no worries about long-term support.”

Airflow checked other boxes, too: It has pre-built integrations with widely used tools like Snowflake and dbt, and it’s a popular solution for working with both. It not only integrates with the most popular ML tools and frameworks, it also complements ML-automation platforms like Kubeflow and MLflow. Just as important, Wilson knew Airflow was a rock-solid orchestration engine that could easily scale to meet Autodesk’s current and future demands.

But Wilson also knew he didn’t want Autodesk taking on the responsibility of setting up, configuring, and running Airflow.

”I had had enough experience with open-source Airflow to know that I didn’t want us managing it ourselves,” says Wilson. “Another person on my team who’d had experience with Astronomer said the company took the pain out of managing Airflow. After some discussion, we felt that with all of the things Astronomer brought to the table, it was the most appropriate choice.”

Astronomer Professional Services Helps with the Heavy Lifting of Migration

Autodesk counted on Astronomer to give it Airflow-powered orchestration that just worked, without the headaches of operating Airflow on its own. But Autodesk also leaned on Astronomer Professional Services for help in migrating its 536 Oozie DAGs to their equivalents in Airflow. Working collaboratively, Autodesk and the Professional Services team came up with a migration plan.

Each day, Wilson, his colleagues, and other Autodesk stakeholders held a 15-30 minute “office hours” session with Astronomer Professional Services, discussing the progress of the migration and talking through any issues that had come up. For the bulk of the day — 4-6 hours — solutions architects and engineers from Professional Services worked with their counterparts at Autodesk, embedded with each of the company’s 25 teams, answering questions, owning problems, and helping teams map their workflows from Oozie to Airflow.

”They collaborated closely with our 25 data engineering teams to help migrate their DAGs, and also to help them learn Airflow, teaching them best practices for data engineering — how to leverage the tools and operators Airflow has,” Wilson says.

Astronomer Professional Services also worked with Autodesk’s engineering teams to help them bootstrap their DataOps development practices. “Our adoption of Astronomer and Airflow is really about helping our teams adopt modern software engineering practices, as well as giving them a self-service infrastructure they can intuitively manage without a big technical barrier,” Wilson explains.

Workflow Orchestration That Just Works

Now that the migration is complete, Wilson says Astronomer’s Airflow-powered orchestration capabilities are already letting Autodesk focus on researching, developing, and delivering new products and services. Product engineers, for example, have the time they need to focus on creative work, and can also iterate much faster when working, accelerating their time-to-delivery.

”We no longer have to spend huge chunks of time fixing outages,” Wilson says. “Our engineering teams can focus their time and resources on product engineering, our analytic engineers and data scientists can focus on getting stakeholders the data they need. We have these modern DataOps software engineering practices, so we can go much faster. We can be way more responsive now.”

The move to Astronomer also empowered Autodesk’s product teams to build, harden, and deploy pipelines on their own, eliminating what Wilson calls a “massive” operational burden.

Astronomer’s open-source Astro CLI has played a big role in this, giving Autodesk’s teams a reproducible local Python development environment that they can use to build, run, and debug their pipelines, as well as deploy them to a separate Airflow deployment for testing.

Once testing is finished, teams can fire up the Astro CLI and deploy their pipelines to production. “With Oozie, our infrastructure and DevOps teams had to deploy code on behalf of engineering teams — it usually wasn’t possible for them to do this on their own,” Wilson explains.

Wilson says one of the best things about Astronomer is that instead of running one or more Airflow versions behind, Autodesk now enjoys same-day access to new versions, as well as built-in support for new features: “From the standpoint of stability and of being on the bleeding edge of what you can do with Airflow, Astronomer is far ahead of any other provider.”

A Smooth Migration

Migrations always involve risk, especially when an organization is re-platforming from software it depends on to support its most business-critical operations. But Wilson was amazed by the uneventfulness of Autodesk’s migration to Airflow and Astronomer: in about 12 weeks, Astronomer Professional Services had moved all of the company’s DAGs from Oozie to Airflow.

”We were able to finish early, which I did not expect at all: I expected us to get about 80-90 percent of the way there and then just have this long tail to clean up. But there was no long tail,” Wilson says.