Enhanced Authentication Security to your Data Services on Azure with Astro
Astronomer is committed to delivering a world-class modern orchestration tool to Azure users, and we’re thrilled to announce a significant enhancement to authentication of your data pipelines running on Apache Airflow™ on Astro - An Azure Native ISV Service to Azure data services. Leveraging modern identity management capabilities provided by Microsoft Entra ID and native integration in Astro, you can realize modern security best practices.
In this blog, we’ll explain the authentication enhancement, our contribution to open-source Apache Airflow, and provide a step-by-step guide to help you get started.
Unlocking Entra ID Workload Identity on Astro
The combination of Microsoft Entra ID (formerly known as Azure AD) platform and Workload Identity capabilities on Azure Kubernetes Service (AKS) natively provide mechanisms to access Entra ID-protected resources, such as Azure Key Vault and Azure SQL.
This new capability on Astro has immense value to both development and security teams, as they can securely access and orchestrate their Azure data services, ensure compliance against their security policies, eliminate the maintenance burden of manually managing credentials, and reduce the risk of leaking secrets or having certificates expire.
In practice on Astro, this requires federated identity credentials to a user-assigned managed identity from Entra ID, that allows your data pipelines to easily and securely access your protected Azure resources. Both the managed identity and federated identity credentials are managed by the Azure platform and do not require you or Astro to provision or rotate any secrets. To learn more about the token exchange workflow between your Astro deployments and managed identities, read about workload identity federation.
Contributing Back to OSS
Astronomer’s longstanding commitment and support for the open-source Apache Airflow project, along with our partnership with Microsoft, has only strengthened through this endeavor and the collaboration with Microsoft on the launch of Astro - An Azure Native ISV Service.
We are pleased to announce that we have added support for Azure Workload Identity authentication to the open-source Airflow Azure Provider via a new DefaultAzureCredential. We believe these new capabilities allow both OSS users and our customers to adopt sound and secure identity management practices for their data pipelines running on Airflow and Azure.
How to Get Started
We are super excited for the adoption of this new capability in Airflow, and even more so on Astro in combination with the recently launched Environment Management feature. Getting started is easy via the Azure Portal. Check out our getting started instructions on our recent blog introducing Astro the latest Azure Native ISV Service.
Once your Organization is created, you’re ready to spin up a new Airflow deployment on Astro with a few simple clicks. You have the option of running your deployment on a Standard multi-tenant Azure cluster or on a Dedicated single-tenant Azure cluster. Check out our Create a Deployment docs for more details.
Within minutes, you’ll have a robust and reliable Airflow deployment running for your data pipelines. To configure Workload Identity, navigate to the Details tab of your deployment and use the link under Advanced > Workload Identity to open the Workload Identity Configuration modal. Enter the information required and follow the instructions to complete the setup in your Azure tenant.
Using Astro’s latest Environment Manage feature, navigate to the Environment tab to create an Airflow connection using your managed identity. You can also do this at the Workspace level and share the connection across multiple deployments.
Note: Use the Managed Identity connection type
Finally, deploy your data pipelines to your deployment on Astro to take advantage of these latest innovations brought to you by Astronomer. Of course, don’t forget to ensure there is network connectivity between your deployment on Astro and the Azure data services your pipelines will be consuming. Check out this doc to learn more about the connectivity options. You can also contact support if you don’t see your preferred connectivity option documented.
Additional Information
Want to dive even deeper into the world of Apache Airflow on Azure with Astro? Watch the on-demand webinar, where our experts walk through the ins and outs, explore practical use cases, and share best practices.
