Astro release notes
Astronomer is committed to continuous delivery of both features and bug fixes to Astro. To keep your team up to date on what's new, this document will provide a regular summary of all changes released to Astro.
Latest Astro CLI Version: 1.24.1 (Release notes)
Latest Astro Runtime Version: 10.6.0 (Release notes)
March 26, 2024
Refactored Astro API documentation
Astro API documentation is now hosted at https://docs.astronomer.io/api. In the new API documentation center, you can:
- Format and test API requests directly in your browser.
- Export requests to Python, Javascript, and curl.
- View weekly changelogs for the API.
Create Deployments with deprecated versions of Astro Runtime
You can now use the Astro API to create Deployments with deprecated versions of Astro Runtime. Using deprecated Astro Runtime versions is sometimes necessary if you're migrating existing Airflow environments to Astro, or if you need to maintain deprecated environments for testing purposes.
Note that this feature is disabled by default. To use this feature, reach out to your account team and request for the feature to be enabled. See Run a deprecated Astro Runtime version for more information.
New GCP database instance types available
You can now use the following node instance types for database instances in GCP clusters:
- XLarge Compute Optimized (24 CPU, 48 GiB MEM)
- XXLarge Compute Optimized (32 CPU, 64 GiB MEM)
See GCP Hybrid cluster settings for a list of all available database instance types.
Additional improvements
- The Astro UI now includes Learning Bytes for features that are not yet configured within your Organization.
- The Astro UI now loads the status for DAG and task runs more quickly.
Bug fixes
- When you change a worker type for an existing worker queue, the Astro UI no longer resets the worker queue's concurrency configurations.
- Fixed an issue where the Astro API did not return the correct value for
IsHibernating
when you queried Deployment information.
March 19, 2024
New Azure regions available on Astro Hosted
You can now create Hosted dedicated clusters in the following Azure regions:
centralus
westus3
southcentralus
See Astro Hosted resource reference for more information.
Custom Deployment roles are now in Public Preview
The ability to customize Deployment-level permissions with Deployment roles is now in Public Preview.
You can additionally use the new Workplace Accessor and Deployment Admin roles to define which users have access to specific Deployments in your Workspace. See more in User permissions reference and and Create and assign custom Deployment roles.
Export data from reporting dashboards using webhooks
In addition to exporting report data with downloads or email, you can now export reporting data using webhooks. Use webhooks to send your reporting data to services such as Segment, Airtable, or Marketo.
Note that webhook exports are a Sigma feature in beta and might experience behavior changes.
Additional improvements
- You can now use the new Credits tab in the Organization Billing page to see your credit balance.
- On Astro Hybrid, the maximum worker concurrency on a worker queue has increased from
64
to256
.
March 7, 2024
Reporting dashboards are now in public preview
Organization dashboards are now in Public Preview to use for examining key metrics across your Organization.
You can also export data from reporting dashboards in the format of your choice. Exports can be triggered on a regular schedule or as an alert when specific criteria are met in your data. Export reporting data to share with other team members or to keep a record of key performance indicators. See Export reporting data for more information.
Customize Deployment-level permissions using Deployment roles
Custom Deployment roles are a new way to define granular permissions for Astro users. For the first time, you can set a user's permissions at the Deployment level and define which specific parts of a Deployment they can access or modify. Use custom Deployment roles to have users collaborate in the same Workspace with only the minimum permissions they require. See Customize Deployment roles for more information.
New Deployment registry cache to improve resiliency
Deployments now include a cache of Astronomer's image registry that stores the current Astro Runtime image for your Deployment. Because Deployments now always have access to their running image, image registry outages should no longer result in failed DAG runs.
Additional improvements
- Removed nonfunctional network usage per Pod metrics from the Deployment Analytics page.
- The end-of-life date for Deployment API keys is June 1, 2024.
- The Cloud UI has been renamed to the Astro UI across all help text and documentation.
- Due to a minor change to Astronomer cluster architecture, you can no longer add custom tags to Hybrid clusters on AWS.
Bug fixes
- Fixed an issue where CPU usage per Pod metrics did not render correctly in the Deployment Analytics page.
- Fixed an issue where the Astro UI didn't show all available Teams when selecting Teams to add to a Workspace.
February 27, 2024
Use a custom service account to authorize Deployments to GCP
You can now attach a custom GCP service account to your Deployment to grant the Deployment all of the service account's permissions to your cloud. Using a custom service account provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing service accounts on new Deployments, or your can attach a single service account to multiple Deployments that should all have the same level of access to your cloud. For setup steps, see Authorize Deployments to your cloud.
Bug fixes
- Fixed an issue where the Astro API failed to list Deployments after you deleted a hibernation override setting.
February 21, 2024
New worker types
Astro Hosted Deployments now support A120 and A160 workers, which include enough CPU and memory to handle the most resource-intensive tasks in your DAGs. See Astro Hosted resource reference for more information about each worker type.
New platform variables to improve Celery executor reliability
Astro Deployments now have the following environment variables set by default. This is to ensure that the Celery executor doesn't freeze when it attempts to connect with a Pod in the Celery backend that has unexpectedly terminated.
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT=5
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE=True
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT=True
For more information about each of these variables, see Platform variables
Ephemeral storage limit on schedulers
Astro now limits that amount of ephemeral storage in a scheduler to 5Gi. If a scheduler attempts to use more than 5Gi of ephemeral storage, it will be terminated.
It is rare for schedulers to require more than 5Gi of ephemeral storage. If your schedulers start to terminate after this update, ensure the following:
- Your Deployment is not producing large amounts of temporary files without cleaning them up.
- If you're using the BingAds Python SDK, your Deployment uses version 13.0.14 or later. Earlier releases include a bug that generates large amounts of temporary files.
Bug fixes
- You can no longer create Teams using the Astro API in an Organization that has SCIM provisioning enabled.
- Astro API responses have been standardized to always return cloud provider names in uppercase (for example,
AWS
).
February 13, 2024
New Astro reporting dashboards show metrics for Deployments across your Organization
This feature is in Private Preview. Please reach out to your customer success manager to enable this feature.
The new Dashboards page includes a suite of dashboards that you can use to asses the performance of Deployments and DAGs across your entire Organization. Each dashboard focuses on a different aspect of your data pipelines to show you opportunities for cost and performance improvements. You can additionally configure Astro to send you alerts when a given metric reaches a specific threshold. See Reporting dashboards for summaries of each available dashboard.
Additional improvements
- When you submit a support request from the Astro UI, you must now define an Active Engagement Period when you or a member of your team can engage with a member of Astronomer support.
- Workspace Members can now access the Clusters view in the Airflow UI for a Deployment.
Bug fixes
- Fixed an issue where network connections between clusters could be disrupted occasionally.
- When you retrieve information about a Deployment through the Astro API, the API now returns an empty value for
EnvironmentVariables
if the Deployment has no environment variables. - Deleting a Workspace through the Astro API now deletes all Astro Cloud IDE projects associated with the Workspace.
- Fixed an issue where you could not clear optional fields in a Deployment's configuration using the Astro API.
February 6, 2024
Bug fixes
- Fixed an issue where all Deployment task logs included the error
Not exporting configs to configmap...
January 30, 2024
New messages for Deployment health status
This feature is in Public Preview.
Astro now automatically monitors Deployments and notifies you when a Deployment isn't running as expected, such as when it can't detect a heartbeat in a scheduler. These notifications, known as Deployment incidents, appear in your Deployment's health status in the Astro UI.
See Deployment health incidents to learn more about each available incident type and how to address them.
Additional improvements
-
You can now access Ask Astro from the Astro UI Help menu:
-
User role titles are now consistently formatted across the Astro UI.
January 25, 2024
Self-service VPC peering and route management for AWS
This feature is in Public Preview.
You can now configure a network connection between Astro and an AWS VPC without contacting Astronomer support. Astro automatically handles creating a connection request and provides instructions for completing the setup yourself. After you create a VPC connection, you can configure routes whenever you need to connect to an additional service in your external VPC. See Create a private connection between Astro and AWS.
Additional improvements
- There is no longer a minimum on the amount of CPU and memory that you can request for Kubernetes Pods.
Bug fixes
- Fixed an issue in Astro Hybrid where refreshing the browser could occasionally reset a worker queue's worker type setting.
January 16, 2024
Additional improvements
- The Astro UI Deployment analytics page now shows CPU Usage Per Pod (%) and Memory Usage Per Pod (MB) as a percentage of your total available resources rather than the resources of a single worker Pod, such that these metrics will never show Deployment resources usage as exceeding 100%.
- The maximum value for worker queue Max # of workers has increased from 30 to 100.
January 9, 2024
Additional improvements
- When you create a Deployment, the Astro UI now shows the Airflow version of your Deployment instead of the equivalent Astro Runtime version.
- Enabled point-in-time restore (PITR) on Hybrid GCP clusters to improve resiliency to outages. Note that this might result in increased costs for cloud storage.
Bug fixes
- Fixed an issue where you occasionally couldn't access the Airflow UI for a Deployment with the error "No healthy upstream".
- Fixed an issue where deleting a user with no Workspace membership from an Organization would affect how their Workspace membership appeared in other, unrelated Organizations.
- Fixed an issue where the Open in Airflow button on the DAG details page in the Astro UI did not open the Airflow UI as expected.
December 20, 2023
Bug fixes
- Fixed an issue where creating an alert for a DAG through the Astro API would apply the alert to the incorrect Deployment.
- Fixed an issue where Deployment worker Pods could crash when running the Kubernetes executor.
- Fixed an issue where Deployment API key expiration dates were not applied correctly if you configured multiple API keys at once.
- Fixed an issue where logging features could be disrupted if you set
AZURE_CLIENT_ID
as an environment variable. Note that this fix applies only to Astro Runtime 10 and later.
December 12, 2023
Bug fixes
- Fixed an issue where the Astro UI would produce an error if you updated an environment variable on an Astro Hybrid Deployment running the Kubernetes Executor.
December 6, 2023
Additional improvements
- The Astro Environment Manager is now generally available. This feature allows you to create and manage Airflow connections in the Astro UI.
Bug fixes
- Fixed an issue where DAG code that appeared in the Airflow UI did not roll back when you rolled back a Deployment, even though the running code was successfully rolled back.
- Fixed an issue where you could not view billing information from the Astro UI when you installed Astro through the Azure Marketplace.
- Fixed an issue where the Astro UI would produce a console error when a user accessed their Workspace list.
November 30, 2023
Support for Microsoft Entra Workload ID
You can now use Microsoft Entra Workload ID to authorize Deployments to resources in Azure. Workload identity is a simple and secure way to authorize access external resources, as it doesn't require creating or storing long-term credentials. To set up Microsoft Entra Workload ID, see Authorize Deployments to cloud resources.
Bug fixes
- Removed the ability to create Hybrid Azure clusters in
eastasia
because some workload identity features aren't supported in this region.
November 16, 2023
Install Astro from the Azure Marketplace
Astro is now available as an Azure Native ISV Service. If your team is considering Astro and you use Azure, Astronomer recommends installing Astro from the Azure Marketplace because:
- You can manage billing from the Azure Portal.
- Microsoft Entra ID is pre-configured for all Organizations.
- It's easier to create Astro resources and get started directly from Azure.
See Install Astro from the Astro marketplace for setup steps. To learn more about Astronomer's partnership with Microsoft, see Introducing Apache Airflow™ on Astro – an Azure Native ISV Service.
Create Airflow connections in the Astro UI and link them to Deployments
You can now create Airflow connections in the Astro UI through the new Environment Manager menu. The Environment Manager lets you create Airflow connections directly in the Astro UI and stores all connections in an Astro-managed secrets backend. You can then share connections between Deployments and set default connections so that your team members always have access to external resources when they create new Deployments. See Create Airflow connections in the Astro UI.
Note that this feature is currently available only for Deployments running the Celery executor.
Trigger a DAG from an Astro alert
You can now configure Astro alerts to trigger any DAG in your Workspace through the Airflow REST API. You can configure the triggered DAG to complete any action, such as sending an alert through a custom communication channel or writing data about the incident to a table.
New Azure regions available on Astro Hosted
You can now create Hosted dedicated clusters in the following Azure regions:
eastus
canadacentral
uksouth
brazilsouth
centralindia
francecentral
japaneast
See Astro Hosted resource reference for more information.
November 7, 2023
Roll back Deployments to previous versions of your code
This feature is in Public Preview.
Astro now maintains snapshots of your past deploys, including your Deployment image and DAG code, for the previous three months. If you need to quickly revert a Deployment back to a working version of your code, you can roll back to a past deploy from the Deploy History page in the Astro UI.
Deploy rollbacks are a powerful safety mechanism to ensure that your production pipelines continue to run when something unexpected happens after a deploy. See Roll back to a past deploy for more information and configuration steps.
Bug fixes
- Fixed an issue where, you could inadvertently open the support request window if you opened a DAG that included "Support" in its name from the DAGs view. As a result of this change, the support request window URL has been updated from
https://cloud.astronomer.io/support
tohttps://cloud.astronomer.io/open-support-request
. - Fixed an issue where the Deployment configuration menu in the Astro UI didn't always show your Deployment's current configuration.
- Fixed an issue where you could not create a Deployment with some stable Runtime versions using the Astro Platform API.
October 31, 2023
Deployment API keys are now deprecated
Deployment API keys have been officially deprecated in favor of Deployment API tokens. This means:
- You can't create new Deployment API keys.
- If a Deployment has no configured Deployment API keys, the API keys tab will not appear.
- You can continue using existing Deployment API keys until a future end-of-support date.
Additional improvements
- Workspace Operators can now create, update, and delete Deployment API tokens.
Bug fixes
- Fixed an issue where a Deployment's Updated By field was not updated if you transferred the Deployment.
October 24, 2023
New Azure regions available on Astro Hosted
You can now create Deployments in standard clusters in the following Azure regions:
eastus2
westus2
westeurope
See Astro Hosted resource reference for more information.
Bug fixes
- Fixed an issue where Deployment API tokens weren't deleted after their associated Deployment was deleted.
- Fixed an issue where you could not create a Deployment with some available Runtime versions using the Astro Platform API.
October 17, 2023
Additional improvements
- You can now view deploy history for both Hosted and Hybrid Astro Deployments in the Astro UI. For more information, see View deploy history.
Bug fixes
- Modified behavior for Astro Hosted so that KubernetesExecutor and KubernetesPodOperator pods running in-cluster have equivalent resource requests and limits. If you do not configure them to have equivalent resource requests and limits, Astro modifies them to the become the limits. Previously, the DAG deploy would fail if
resources
did not equallimits
.
October 10, 2023
Deployment API tokens
Deployment API tokens are now generally available and replace Deployment API keys as the most secure and customizable way to programmatically update Deployments. This includes using them to deploy code and update environment variables.
See Deployment API tokens to learn how to create and manage Deployment API tokens.
Deployment API tokens are a direct replacement for Deployment API keys, which are now supported only on a limited basis on Astro.
After October 31, 2023, you will not be able to create new API keys. While you can still continue to use and manage existing Deployment API keys, Astronomer will soon require using Deployment API tokens.
New Edit Deployments in the Astro UI
Editing Deployments in the Astro UI has a new, consolidated flow. All of the configuration options are now editable in a single form, similar to Deployment creation, instead of spread across multiple forms. See Deployment Settings for a detailed description of how to create, update, and configure your Deployment options.
October 3, 2023
Additional Improvements
- Added a DAG Success alert so you can now set up an alert for successful completion events. See how to set up Astro alerts.
Bug Fixes
- Fixed a problem in the Astro UI where a warning about Deployment Health was displayed when a Workspace had zero Deployments.
September 26, 2023
Introducing the Astro API
The Astro API is currently in beta. See Astro API versioning and support.
You can now use the Astro API to create applications and scripts to programmatically interact with Astro. The Astro API is a standard REST API that includes endpoints for interacting with all key resources and components on Astro.
Using the Astro API, you can create robust and secure applications for managing Deployment resources, updating user permissions, and performing many other key Astro operations. To make your first API call, see Get started with the Astro API.
September 19, 2023
Manage Deployments programmatically using Deployment API tokens
Deployment API tokens replace Deployment API keys as the most secure and customizable way to manage Deployments programmatically. You can use Deployment API tokens to perform all of the same actions as a Deployment API key, including:
- Pushing code to a Deployment.
- Updating a Deployment's environment variables.
- Making requests to update your Deployment's Airflow environment using the Airflow REST API.
Unlike Deployment API keys, you can set an expiration date for Deployment API tokens and rotate them to better manage access to your Deployment. See Deployment API tokens to learn how to create and manage Deployment API tokens.
Deployment API tokens are a direct replacement for Deployment API keys. Therefore, Astronomer recommends always using Deployment API tokens over API keys. While you can still continue to use and manage existing Deployment API keys, Astronomer will soon require using Deployment API tokens.
After API tokens are generally available, Deployments with zero API keys will not show the API Keys tab and you will no longer be able to create Deployment API keys. If you want to continue using API keys, ensure that you always have at least one API key configured for the Deployment.
Additional improvements
- When you create a new Deployment, the Astro UI now presents new options and suggestions for running your first DAG.
- You can now retrieve a Workspace's ID from the Astro UI. To find a Workspace's ID, open the Workspace in the Astro UI and go to Workspace Settings > General.
September 12, 2023
Per-Deployment IAM workload identities on AWS
Astro Hybrid clusters on AWS now support per-Deployment IAM workload identities, meaning that you can now limit your trust policies to authorize only specific Deployments to your cloud resources.
This change required an automatic update to the cross-account role that Astro uses to manage clusters in your cloud. In addition to enabling per-Deployment IAM workload identities, this update also adds the following permissions to reduce the risk of partial deletions in your cloud:
{ "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DeleteLoadBalancer" }
For more information about this change, see Automatic updates coming to cross-account roles for Astro Hybrid on AWS.
To migrate from using cluster workload identities to Deployment workload identities:
-
In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard. Identify all of your trust policies that specify your cluster workload identity. They should look similar to the following trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<dataplane-AWS-account-ID>:role/AirflowS3Logs-<cluster-ID>"
]
},
"Action": "sts:AssumeRole"
}
]
} -
For each trust policy, add the workload identities for any Deployments that you want to access the related resource. To locate your Deployment workload identity, open the Deployment in the Astro UI and copy the Workload Identity from the Details page. Your trust policy should now look like the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::123456789876:role/AirflowS3Logs-cl6zcnlc641hr0voibivf21jh",
"arn:aws:iam::<dataplane-AWS-account-ID>:role/astro-<namespace>"
]
},
"Action": "sts:AssumeRole"
}
]
} -
For each Deployment that you specified in your trust policies, open the Deployment in the Astro UI and click Details, then click Edit Details. In the Workload Identity section, select the new Deployment identity from the dropdown list, then click Update. To avoid disruption to tasks, do not complete this step until you have added the Deployment workload identity to all of the trust policies it needs for access.
-
Upgrade to the latest Astro CLI release, which includes support for Per Deployment IAM Workload Identity.
-
After you've tested the policies with your Deployment workload identities, remove the cluster workload identity from your trust policies
Bug fixes
- Fixed an issue where Billing Admins could view task usage on the Usage page only for Workspaces that they belonged to. Now, Billing Admins can view usage for all Workspaces regardless of their Workspace role.
Additional improvements
- The Astro UI Usage page now shows task usage for deleted Deployments. If you're an Astro Hybrid Billing Admin, this means that task usage metrics now better reflect your billable usage.
- When you create a Deployment through the Astro UI and choose an Astro Runtime version, you can now select only the most recent supported patch for each major version of Astro Runtime.
- You can now filter task logs by log level or source from the DAGs page in the Astro UI.
September 6, 2023
View deploy history in the Astro UI
When you view a Deployment in the Astro UI, you can now open the Deploy History tab to view a table of all code deploys. The table shows who made deploys, when they made the deploys, and what Astro Runtime image they used for the deploy.
You can also now use the Astro CLI to specify an optional description for your deploys using the --description
flag. Deploy descriptions appear in the Deploy History table and are useful for telling other Workspace members why you made a deploy or what changes it contains. For more information, see View deploy history.
View rendered data figures in the Astro Cloud IDE
Python cells that generate figures using Matplotlib, Plotly, or any libraries that extend these tools now show the figures they render in the Astro Cloud IDE. When you run a Python cell, any figures it generates appear in a new Figures tab.
August 29, 2023
Changes to Workspace user roles
To take advantage of these new user roles programmatically, you must upgrade the Astro CLI to version 1.19 or later.
To increase granularity and better serve each user persona on Astro, Workspace roles have been updated with new names and permissions:
- The Workspace Author role is a new role for users who primarily write and deploy DAGs. Users with this role can push code changes, but they can't update Deployment or Airflow settings such as Airflow variables, Astro environment variables, or connections.
- The Workspace Admin role has been renamed to Workspace Owner. Users with this role are responsible for administrating membership to the Workspace.
- The Workspace Editor role has been renamed to Workspace Operator. In addition to pushing code changes, Workspace Editors can now also edit Airflow objects such as variables, connections, and XComs. Users with this role are responsible for managing the environments that DAGs run in.
- The Workspace Viewer role has been renamed to Workspace Member. Users with this role only need viewing permissions for a Deployment and don't have permissions to make any code or configuration changes.
For more information about these role changes, see User permissions reference and Enhanced Astro Workspace Roles for more granular permissions
Additional improvements
- The lifespan of the personal user access token you can retrieve from
cloud.astronomer.io/token
has been reduced from 24 hours to 1 hour. - The DAGs view of the Astro UI now shows your configured dependency edge labels in the graph view.
- The Astro UI now shows more detailed instructions for deploying code when you create a new Deployment.
- The Deployment Analytics page in the Astro UI has been renamed to Overview.
Bug fixes
- Fixed an issue where Deployments using the Kubernetes executor could not run DAGs with lower resource requests than the Default Pod Size. Minimum requests are now hard-coded and decoupled from default requests.
August 21, 2023
Additional improvements
- You can now configure task log forwarding to Datadog at the Deployment level.
- In the DAGs view of the Astro UI, you can now double click a task run node in the graph view to view the task run's logs and mapped tasks.
- The A50 worker type has been renamed to A60 to make it consistent in scale with other worker types.
- The max possible CPU quota and Memory quota for a Deployment running in a Hosted dedicated cluster has increased to 1600 vCPU/ 3200 GiB respectively.
August 15, 2023
Additional improvements
- You can now see how many Astro alerts you've configured for a DAG in the DAGs page of the Astro UI.
Bug fixes
- Fixed an issue where you couldn't run Astro Cloud IDE pipelines that included a Markdown cell.
- Fixed an issue where an Organization's SSO bypass link was formatted incorrectly in the Astro UI.
August 8, 2023
New Hosted worker type
You can now configure Deployments with the A50
machine type, which has 12 vCPU and 24 GiB. See Astro hosted resource reference.
Additional improvements
- When you create a new Astro Cloud IDE project, you can now specify whether you want the project to include an example pipeline.
- You can now access Organization-level settings in the Astro UI only through the Organization Settings link. Additionally, some Organization settings have been moved to the top level of navigation so that there is no longer a Settings menu.
- You can now use commas, apostrophes, and ampersands in Workspace and Organization names.
- The Workspace list view in the Astro UI has been redesigned so that Organization Owners can now edit and delete Workspaces directly from the list.
Bug fixes
- Fixed an issue where the Astro UI showed incorrect CPU and memory limits in bar charts on the Deployments list and Details page.
August 1, 2023
Hosted Deployments have DAG-only deploys enabled by default
New Astro Hosted Deployments now have DAG-only deploys enabled by default. When DAG-only deploys are enabled, some workflows for your Deployment, including image-based deploys, are different compared to when DAG-only deploys are disabled. For more information about how code and image deploys work when DAG-only deploys are enabled, see What happens during a project deploy. To disable DAG-only deploys, see Enable/ disable DAG-only deploys on a Deployment.
Teams now have Organization-level roles
All Teams on Astro now have an Organization role. Existing Teams have been given the Organization Member role, which doesn't result in any additional automatic permissions.
Coupled with SCIM user groups, you can now manage your Organization Owners and Billing Admins from your identity provider. See Manage teams for more information.
Additional improvements
- The Astro UI now shows how many Workspaces, DAGs, clusters, and Astro Cloud IDE projects you have in the left sidebar.
- You can now create Deployments in standard clusters hosted in
europe-west2
on GCP andeu-central-1
on AWS. - The default metadata database instance type for new Deployments on GCP clusters in Astro Hybrid has been reduced to
Small General Purpose
with 2 vCPUs and 8GiB. See GCP Hybrid cluster settings.
Bug fixes
-
Because some regions don't support specific machine types that Astro Hosted uses, you can no longer create Hosted dedicated clusters in the following AWS regions:
af-south-1
ap-east-1
ap-northeast-2
ap-northeast-3
ca-central-1
eu-south-1
eu-west-3
me-south-1
July 25, 2023
Additional improvements
- The templates for Astro alert messages have been updated to include more information about the Deployment that the alert was triggered in, including a link to the DAG that triggered the alert.
Bug fixes
- Fixed an issue where Azure AD single sign-on (SSO) connections were incorrectly labeled as SAML connections in the Astro UI.
July 18, 2023
Configure default Pod sizes
You can now configure the default minimum CPU and memory for tasks that you run with the Kubernetes executor or KubernetesPodOperator. If you don't specify CPU or memory in a task definition, Astro runs the task in a Pod that uses your default resource configurations. Configure default minimum resources to ensure that tasks always have enough CPU and memory to run successfully. See Deployment settings.
New regions available on Astro Hosted
You can now create Hosted dedicated clusters in the following regions:
-
AWS
af-south-1
- Africa (Cape Town)ap-east-1
- Asia Pacific (Hong Kong)ap-northeast-1
- Asia Pacific (Tokyo)ap-northeast-2
- Asia Pacific (Seoul)ap-northeast-3
- Asia Pacific (Osaka)ap-southeast-1
- Asia Pacific (Singapore)ap-southeast-2
- Asia Pacific (Sydney)ap-south-1
- Asia Pacific (Mumbai)ca-central-1
- Canada (Central)eu-central-1
- Europe (Frankfurt)eu-south-1
- Europe (Milan)eu-west-1
- Europe (Ireland)eu-west-2
- Europe (London)eu-west-3
- Europe (Paris)me-south-1
- Middle East (Bahrain)sa-east-1
- South America (São Paulo)us-east-1
- US East (N. Virginia)us-east-2
- US East (Ohio)us-west-1
- US West (N. California)us-west-2
- US West (Oregon)
-
GCP
asia-east1
- Taiwan, Asiaasia-northeast1
- Tokyo, Asiaasia-northeast2
- Osaka, Asiaasia-northeast3
- Seoul, Asiaasia-south1
- Mumbai, Asiaasia-south2
- Delhi, Asiaasia-southeast1
- Singapore, Asiaasia-southeast2
- Jakarta, Asiaaustralia-southeast1
- Sydney, Australiaaustralia-southeast2
- Melbourne, Australiaeurope-central2
- Warsaw, Europeeurope-north1
- Finland, Europeeurope-southwest1
- Madrid, Europeeurope-west1
- Belgium, Europeeurope-west2
- England, Europeeurope-west3
- Frankfurt, Europeeurope-west4
- Netherlands, Europeeurope-west6
- Zurich, Europeeurope-west8
- Milan, Europeeurope-west9
- Paris, Europenorthamerica-northeast1
- Montreal, North Americanorthamerica-northeast2
- Toronto, North Americasouthamerica-east1
- Sau Paolo, South Americasouthamerica-west1
- Santiago, South Americaus-central1
- Iowa, North Americaus-east1
- South Carolina, North Americaus-east4
- Virginia, North Americaus-east5
- Columbus, North Americaus-south1
- Dallas, North Americaus-west1
- Oregon, North Americaus-west2
- Los Angeles, North Americaus-west3
- Salt Lake City, North Americaus-west4
- Nevada, North America
See Astro Hosted resource reference for all available configurations.
Bug fixes
- You can no longer create Hybrid GCP clusters in
eu-north-1
.
July 11, 2023
Send Astro alerts to email
You can now send Astro alerts to multiple email addresses. Sending Astro alerts to email requires no configuration outside of Astro, which makes it a quick option to improve your alerting infrastructure. See Configure Astro alerts for setup steps.
Configure SCIM provisioning for Azure AD
If your Organization uses Azure for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See Set up SCIM provisioning for more information.
Additional improvements
- On Astro Hosted deployments, the
astronomer_monitoring_dag
has been paused for all image-based Deployments and removed entirely from all Deployments with DAG deploys enabled. It has been replaced with an implementation that allows workers on Deployments to fully scale to 0.
July 5, 2023
Configure SCIM provisioning for Okta
If your Organization uses Okta for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See Set up SCIM provisioning for more information.
See pricing estimate when creating a Deployment
The Deployment creation page in the Astro UI has been reorganized to make it easier to focus on specific configurations for your Deployment. Each configuration is now collapsible and includes guidance for different environment sizes. Additionally, the page now shows cost estimates for a Deployment before you create it.
Additional improvements
- You can now configure Deployments with the
A40
machine type, which has 8 vCPU and 16 GiB. See Astro hosted resource reference. - You can now access your Organization settings from a Workspace by clicking the name of your Organization/ Workspace.
- On Astro Hybrid, the default Azure DB instance is now
Standard D2ds_v4
. - The Deployment creation screen for Astro Hybrid has received several updates that were previously only available on Astro Hosted.
Bug fixes
- Fixed an issue where DAGs that used the KubernetesPodOperator and had tasks with
in_cluster=False
could not be parsed.
June 27, 2023
Support for dedicated clusters on Azure
You can now create a dedicated cluster in the following Azure regions:
australiaeast
eastus2
northeurope
westeurope
uswest2
See Astro Hosted resource reference for more information.
Additional improvements
- The Astro UI now shows how many Workspaces each Team belongs to in Settings > Access Management > Teams.
- You can now create dedicated clusters in
us-west1
on GCP.
June 20, 2023
Additional improvements
- You can now add a new Astro user to Workspaces before the user has accepted their invite.
- If you have Organization Owner permissions, you can now add a user to a Workspace even if the user hasn't been added to your Organization. Users added to Workspaces this way are automatically added to your Organization as an Organization Member.
- The Astro UI now shows your Team IDs in Settings > Access Management > Teams. Use Team IDs to add Teams to Workspaces using the Astro CLI.
- A Team's Updated At and Updated By values are now updated when you change the Team's permissions in a Workspace or Organization.
Bug fixes
- Fixed an issue where a Workspace descriptions were incorrectly required when creating a new Workspace through the Astro CLI.
June 13, 2023
Manage billing and track usage for Astro Hosted
Use the new Billing page in the Astro UI to see both high-level and detailed metrics about your spend in Astro Hosted. You can also use this page to configure your billing details and view invoices. See Manage billing for more details.
New cell type for using Airflow operators in the Astro Cloud IDE
You can now use any Airflow operator available on the Astronomer Registry in your Astro Cloud IDE pipeline. Operator cells apply formatting and checks for parameter inputs, making it easy to configure operators as part of your pipeline. See Use Airflow operators in the Astro Cloud IDE.
Additionally, you can configure custom cells to use your team's custom operators in a pipeline. See Create custom operator cells.
IMDSv2 is now enforced on AWS clusters
If your DAGs assume IAM roles to directly access metadata on your cluster using IMDSv1, this change can result in DAG run failures. Upgrade your DAGs to use IMDSv2 for all cluster metadata requests.
See Use IMDSv2 for more information.
Astronomer now enforces IMDSv2 on all AWS clusters. Any requests for resources on your clusters are now session-based and muse include a token in the request body.
Additional improvements
- Trial Deployments now have DAG-only deploys enabled by default.
- The Astro UI now shows your Organization Short Name and Astro SAML Connection Name in the Astro UI.
- You can now view mapped tasks from the DAGs page in the Astro UI.
Bug fixes
- Fixed an issue where worker node pools in Hosted dedicated clusters on Azure were not being updated correctly.
- Fixed an issue the Astro UI would reset a Deployment's Min Worker Count from 0 to 1 after you edited the Deployment in any way.
June 6, 2023
Track user actions and ensure compliance with audit logs
You can now export audit logs from the Astro UI to view all actions taken in your Organization over a given time period. See Export audit logs for setup steps.
Additional improvements
- You can now configure Hybrid GCP clusters with additional Memory Optimized and Compute Optimized Cloud SQL instance types. See Supported Cloud SQL instance types.
May 30, 2023
Manage permissions for groups of users with Teams
Configure Teams from the Astro UI to manage the permissions for many users across Workspaces from a single page. Teams are a group of users in an Organization that you grant the same Workspace permissions, without needing to define them individually.
See Make a Team for setup steps.
Bug fixes
- In Astro Hosted, an irrelevant AWS external ID info page has been removed from the Astro UI.
- Fixed an issue where DAG-only deploys could be unreliable due to the deploy process not requesting enough resources in the cluster.
May 23, 2023
Introducing Astro Hosted and Hybrid
Astro Hosted is a new way to run Airflow on Astronomer's cloud. On Astro Hosted, Airflow environments are managed and hosted entirely by Astronomer, enabling you to shift your focus from infrastructure to data.
For more information about how Astro Hosted works, see the Architecture overview.
If you're already an Astro user and your Deployments run in your company's own cloud, you're using Astro Hybrid. This version of Astro was formerly known as Astro - Bring Your Own Cloud.
To see whether you're an Astro Hybrid user, open your Organization in the Astro UI and go to Settings > General. Your version of Astro is listed under Product Type.
See Documentation refactor for Astro Hybrid to learn how the documentation has changed for current Astro Hybrid users.
Configure default Kubernetes Pods on Astro Hosted
One of the biggest risks of running the Kubernetes executor or KubernetesPodOperator is that your tasks can accidentally request more resources than expected, which can drive up costs. To limit this risk, you can now configure default and maximum Pod resources from the Astro UI. If a task tries to request Pod resources that are more than your configured limits, the task fails.
See Configure Kubernetes Pod resources for setup steps.
Documentation refactor for Astro Hybrid
The following updates have been made to documentation to accommodate new Astro Hosted information:
- Astro Hybrid documentation now has a dedicated menu under "Administration" that contains all docs related to Hybrid installation and cluster management. See Astro Hybrid overview.
- All other docs now assume Astro Hosted by default. If a feature is functionally different between Hosted and Hybrid, the documentation for that feature will include a note about how that setup differs for Hybrid. Look for the blue Alternative Astro Hybrid setup notes throughout documentation.
May 16, 2023
Automate Organization management with Organization API tokens
You can now create Organization API tokens to automate key actions across your Organization and all of the Workspaces in it. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes. Some common actions that you can automate with Organization API token are:
- Creating Workspaces.
- Inviting users to an Organization or Workspace.
- Creating and updating Deployments using a Deployment file.
- Exporting audit logs.
- Gathering metadata about Deployments using the Airflow REST API.
- Completing any of the actions you can complete with a Workspace API token or Deployment API key across all Deployments in your Organization.
See Manage Organization API tokens for more information.
May 2, 2023
Receive Astro alerts on Slack or PagerDuty
Astro alerts are a new way to be notified when your DAGs aren't running as expected. Unlike Airflow callbacks and SLAs, Astro alerts require no changes to DAG code and integrate with Slack and PagerDuty.
You can set an alert on any DAG to be notified when the DAG fails or when a task takes longer to run than expected. See Astro alerts for configuration steps.
Bug fixes
- Fixed an issue where SSO configurations made through Astronomer support could be overridden by updating the SSO configuration through the Astro UI.
April 26, 2023
Improved log viewing in the Astro UI
The Deployment Logs page in the Astro UI now shows logs for your Deployment's workers, schedulers, triggerers, and webserver. Additionally, you can now view up to the last 10,000 logs emitted by your Deployment from the Astro UI.
To make it easier to parse this larger log volume, the Logs page now lets you filter by log type, date, and keyword. See View logs for more information.
April 18, 2023
Self-service configuration for single sign-on (SSO) connections
You can now configure SSO connections directly from the Astro UI without assistance from Astronomer support. Use the Authentication page to configure different authentication environments for your Organization by creating and managing multiple SSO connections and domains.
To review the new process for creating SSO connections, see Set up authentication and SSO. To create new managed domains to map to your SSO connections, see Manage domains.
April 11, 2023
Additional improvements
- The node type for running Airflow system components on GCP clusters has been reduced from
n2-standard-4
toe2-standard-4
. - To optimize infrastructure costs for running the Kubernetes executor, Kubernetes executor worker Pods from different Deployments can now run on the same worker node. This occurs only when the Deployments are hosted in the same cluster and use the same worker node instance type.
April 4, 2023
Preview Deployments
You can now create preview Deployments from feature branches in your Git repository. Use a preview Deployment template or GitHub Actions template to configure your Astro pipelines to:
- Create the preview Deployment when you create a new branch.
- Deploy code changes to Astro when you make updates in the branch.
- Delete the preview Deployment when you delete the branch.
- Deploy your changes to your base Deployment after you merge your changes into your main branch.
Additional improvements
- Added the ability to enforce CI/CD deploys. You can now configure your Deployment to only accept code deploys if they are triggered by a Deployment API key or Workspace token.
- When you create a new cell in the Astro Cloud IDE, the editor auto-scrolls to your new cell and selects it.
Bug fixes
- Fixed a bug where the UI passed the wrong cluster type.
- Fixed an issue where the Deployment status shows as 'deploying' when KPOs are running.
March 28, 2023
New GCP node instance types available
You can now use the following node instance types for worker nodes in GCP clusters:
e2-standard-32
e2-highcpu-32
n2-standard-32
n2-standard-48
n2-standard-64
n2-highmem-32
n2-highmem-48
n2-highmem-64
n2-highcpu-32
n2-highcpu-48
n2-highcpu-64
For a list of all instance types available for GCP, see Supported worker node pool instance types.
Additional improvements
- You can now use
db.m6g
anddb.r6g
RDS instance types on AWS clusters. - The default RDS instance type for new AWS clusters has been reduced from
db.r5.large
todb.m6g.large
- The default CIDR range for new AWS clusters has been reduced from /19 to /20.
- You can now submit a Request type in the Astro UI support form. When you choose a request type, the form updates to help you submit the most relevant information for your support request.
- You can no longer delete a Workspace if there are any Astro Cloud IDE projects still in the Workspace.
- Organization role permissions have changed so that only Organization Owners can create Workspaces.
Bug fixes
- Fixed an issue where you could set a Deployment's scheduler resources to less than 5 AU.
March 21, 2023
Automate Workspace and Deployment actions using Workspace API tokens
Use Workspace API tokens to automate Workspace actions, such as adding users to a Workspace and creating new Deployments, or for processes that a Deployment API key can automate. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes.
To create and use Workspace API tokens, see Workspace API tokens.
Additional improvements
-
In the Astro Cloud IDE, you can now specify the output table for a Warehouse SQL cell using both literal and Python expressions. See Create a SQL cell.
-
Port 80 is no longer used for certificate management on the data plane.
-
To switch Organizations in the Astro UI, you now use the Switch Organization button next to your Organization's name.
March 15, 2023
Run the Kubernetes executor in Astro
You can now configure your Deployments to use the Kubernetes executor for executing tasks. Using the Kubernetes executor, you can:
- Ensure that tasks running longer than 24 hours are not interrupted when your team deploys code.
- Run tasks with different version dependencies in the same Astro project.
- Request specific amounts of CPU and memory for individual tasks.
- Automatically down your resources when no tasks are running.
The Kubernetes executor runs each task in its own Kubernetes Pod instead of in shared Celery workers. Astronomer fully manages the infrastructure required to run the executor and automatically spins Pods up and down for each of your task runs. This executor is a good fit for teams that want fine-grained control over the execution environment for each of their tasks.
To learn whether the Kubernetes executor works for your use case, see Choose an executor. To configure the Kubernetes executor for a task or Deployment, see Configure the Kubernetes executor.
Simplified Organization management in the Astro UI
The Astro UI has been redesigned so that Organization settings tabs are now available in the left menu. Use this new menu to switch between pages as you can for Workspace settings.
While most tabs were migrated directly to the left menu with the same name, some pages have been renamed and moved:
- Formerly located in Overview, your Workspace list is now available in Workspaces.
- Formerly located in the People tab, Organization user management settings are now in Settings > Access Management.
- Formerly located in the Settings tab, general Organization settings are now in Settings > General.
New Astro Cloud IDE integration with GitLab
You can now configure a GitLab repository in your Astro Cloud IDE project. Configuring a GitLab repository allows you to commit your pipelines and deploy them to Astro directly from the Astro Cloud IDE. See Deploy a project from a Git repository to Astro.
Additional improvements
- Clusters on an Astro - Hosted installation no longer retain Airflow logs which are older than 90 days.
- The Data Plane System node pool instance type on GCP clusters has been reduced from
n2-standard-4
ton2-standard-2
.
March 7, 2023
Get expert advice on Astro and Airflow in office hours
Office hours are a new way for Astro customers to meet with the Astronomer Data Engineering team. In an office hour meeting, you can ask questions, make feature requests, or get expert advice for your data pipelines.
You can now schedule a 30-minute office hour meeting in the Help menu next to your user profile in the Astro UI.
For more information, see Book office hours in the Astro UI.
Additional improvements
- The node pool instance type used for Astro system components on GCP clusters has been reduced from
n2-standard-4
ton2-standard-2
. - DAGs generated by the Astro Cloud IDE now use UTC instead of your current timezone as the default timezone for scheduling DAG runs.
Bug fixes
- Fixed an issue in the Astro Cloud IDE where you could not update a pipeline that was configured with an invalid cyclic dependency chain.
- Fixed an issue where deploying an Astro project with a custom Docker image tag resulted in the Deployment always having the Deploying status in the Astro UI.
- Fixed an issue where worker Pods on Azure clusters were sometimes unable to scale because there was no prioritization for starting up essential scheduling Pods.