2024 Astro release notes

​

Additional improvements

• Two new Astro alert types, Task Failure and DAG Duration are generally available.
• Add icon to certain Deployment fields to indicate that they cannot be edited after the Deployment is created.

​

Bug fixes

• Fixed a bug that prevented users with custom roles at their Team level from making API tokens with the same level of permissions as themselves.

​

Additional improvements

• Added bundle as a deploy type for Deploy APIs to support dbt deploys via the Astro API.

​

Pay-as-you-go Astro now available on AWS Marketplace

​

Additional improvements

• Two new Airflow application metrics are available for the Universal Metrics Exporter, dagrun.first_task_scheduling_delay and task_instance_created_<operator_name>. See Export metrics for more information.
• Improved Astro’s logic for choosing Deployment fallback contact emails.
• Added message in Deployment details to clarify if the contact emails listed are a fallback value or not.

​

Bug fixes

• Fixed an issue where Data Product Alerts were unexpectedly showing up when using the Workspace filter in the Organization Alert Management List.
• Fixed a bug where asset filters weren’t incorporating the entered search.

​

Additional improvements

• dbt deploy is now generally available.
• Improved error messaging when an Organization Owner email is not supplied, preventing you from creating default Deployment alerts.
• Updated the onboarding flow so that newly invited users see a signup page, instead of the login page.
• Added the ability to retrieve audit logs via the Astro API. See Audit logs reference and Astro API changelog for more information.

​

Bug fixes

• Fixed a bug in Observe where selecting a dataset with only downstream dependencies did not fully update the asset lineage graph. Now, all dependencies are correctly displayed.

​

Network egress management

​

Additional improvements

• Added the ability to work with the Astro Environment Manager through the Astro API, by working with Environment Objects. Environment Objects are the API representation of the different functionalities the Environment Manager supports such as Universal Metrics Export, Airflow Variables, and Connections.
• Added the ability to filter assets list by asset type, namespace, and dag.
• Updated Deployment configuration page.
• Removed Airflow Variable Key from UI when editing Airflow Variable.
• Moved the IP Access List from being a sub-section of the Authentication settings page to a dedicated page within Organization Access Management.

​

Bug fixes

• Fixed a bug where only Organization owners were considered as fallback emails for Deployment Health Alerts alerts. With this fix, Workspace owners will be considered first and Organization owners will only be considered as a final fallback if no human Workspace owners exist.

​

Proactive Alerts for Deployment health

​

Create data products with Astro Observe

​

Additional improvements

• Added validation of environment variable keys in deployment create and update requests. Environment variables must now match the following regex: ^[a-zA-Z_]+[a-zA-Z0-9_]*$.
• Upgraded Azure clusters to Kubernetes 1.29
• Improved descriptions for scheduler sizes and High Availability in advanced section of cluster creation.
• Improved description of Development Mode in advanced section of Deployment creation.
• Removed a warning that displayed in the Astro UI when a user’s Deployment was using CI/CD Enforcement.

​

Bug fixes

• Fixed an issue where Open Airflow button was disabled, even when the Deployment was not getting created or hibernating.
• Fixed a bug where the Airflow webserver would restart for dag-only deploys.

​

Airflow mini-scheduler turned off by default

​

Bug fixes

• Fixed a bug where only users with Organization-scoped roles could create or update Deployment alerts, even if they had Deployment alert creation and update permissions enabled.

​

Bug fixes

• Fixed a bug affecting dbt deploys for Deployments using the Kubernetes executor.

​

Additional improvements

• The Scope and Scope Entity ID columns of the Alerts and Notification Channels lists are now consolidated to the Scope column, which displays the entity name and links to the Deployment or Workspace the alert is enabled for. If the scope of the Alert or Notification Channel is the entire Organization, the Scope column value is This Organization. This enhancement allows you to navigate directly to the Deployment or Workspace from the list.

​

Bug fixes

• Fixed a bug where the Default Pod Size field was showing incorrect sizes.

​

Updates and changes to Astro Alerting UI

• You can now define whether a notification channel is available to specific Deployments, an entire Workspace, or to an entire Organization.
• You can create and edit alerts and notification channels of any type through the Organization Alerting page, whether they are Deployment or Organization alerts. If you don’t have user permissions with privileges to make a particular type of alert, you can still access the Alerting editor in the Organization menu to create, edit, and manage both alerts and notification channels.
• Previously, you could access and edit Workspace alerts from the Alerting page in the main navigation of the Astro UI. Now, you can create and manage your Organization alerts through the Alerting page in the Organization menu, or your Deployment-specific alerts in your Deployment’s Alerts page.

​

Bug fixes

• Fixed a bug where editing Airflow variable links in the Environment Manager would delete any links to Deployments. Now, editing Airflow variables in the Environment Manager does not affect linked Deployments.

​

Additional improvements

• Added Updated By and Created By columns to the Workspaces list.
• Enhanced the onboarding experience so you can use one of the template Deployments without connecting to your GitHub account.

​

Bug fixes

• Fixed an issue that showed incorrect pagination numbers in the table footer when searching the Alerts list.
• Resolved an issue where the globalFilter search paramameter would not clear if you deleted the input value entirely.
• Fixed an issue with the dbt deploy rollback functionality where dbt bundles weren’t updated to the rollback deploy version’s bundle state.

​

Additional improvements

• The Universal Metrics Exporter is now generally available to Team-tier level customers. See Export metrics for information about how to export infrastructure metrics about your Airflow use on Astro to your preferred third-party observability tools.
• Updated the Alerts tab in the Astro UI to improve bulk-selecting alerts, increase the number of characters you can use when naming an alert, and allow auto-generated alert names to include the alert type in the alert name.
• Improved the display of long strings to have a truncated display, which expands to show the full string when you hover over or copy the value.

​

Bug fixes

• Fixed an issue where you couldn’t submit Log Summary feedback for different task runs that had the same task ID.

​

Additional improvements

• Updated the Organization Dashboards’ Cost Breakdown tab, which now displays top-level Network and Cluster costs.

​

Bug fixes

• Fixed an issue which prevented KPO task Pods from terminating correctly.

​

Scale Deployments Reliably with XL Deployment size and standalone Dag processor

​

Astro UI improvements

• A new section in the main menu to include an Organization section with links to important resources for administering your Astro Organization. These include links to your Organization Settings, Organization Dashboards, and quick access to your cluster settings.
• A dedicated Deployment Analytics page in your detailed Deployment information, which now includes improvements to the tooltips and metrics visualization design. See Deployment metrics for more information about the available metrics.
• Your Deploy History now appears on the Overview page of your Deployment. See Deploy history for more information about how to view your code deploy history.

​

Create and manage Airflow variables with the Environment Manager

​

Additional improvements

• Made updates to the Astro UI for editing Astro Alerts, to clarify whether you are discarding your edits to an alert, or discarding the alert.

​

Bug fixes

• Fixed an issue that blocked some users from pushing project images to the Docker registry.

​

Additional improvements

• You can now give Airflow Dataset Create and Delete permissions to custom roles. This capability is provided to Deployment Admin, Workspace Author, Workspace Operator, and Workspace Owner default roles. Previously only Workspace Owner and above had access to this permission set. See more information about User Permissions and Custom Deployment Roles.
• If your Astro Deployments run on AWS or GCP clusters, you can now see the Account ID of the clusters in the Cluster Details page in your Organization Settings.
• For Deployment alert emails, the sender now shows as Astro. It previously showed the sender as Postmaster.
• You can now see Deployment details while Astro creates your Deployment, where only the specific actions that are not available until after creation is complete are disabled.

​

Additional improvements

• Added the ability to specify Custom Workload Identity at Deployment creation time when creating an AWS Deployment.
• Improved the descriptions of default hibernation schedules in the UI.
• Added more detail in descriptions for code deploy failures in the Deploy History page.

​

Bug fixes

• Fixed an issue where you could not use global text search for dags in your Alerts.
• Resolved a problem where search query parameters might be deleted when using pagination.

​

Additional improvements

• Improved the descriptions of Deployment Health Alerts to provide more specific information about performance thresholds.

​

Bug fixes

• Fixed an issue where dag lists didn’t update when you switched Deployments in the Astro UI.
• Fixed a bug where the cursor would disappear while typing in certain fields in the Astro UI.

​

Customize labels on your metrics export

​

Additional improvements

• Added a new template demonstrating how to run dbt projects on Astro using Cosmos when you’re onboarding. See Get started on Astro to set up a trial account and try out a template.

​

Bug fixes

• Fixed an issue where a user could not bulk select alerts if they had different names but the same configurations.

​

New alerts for Deployment health incidents

• Improve alerting coverage beyond dag and task failures to address infrastructure-level incidents that are otherwise difficult to monitor.
• Proactively monitor Deployment health and take immediate remediation actions through email, Slack, and PagerDuty to reduce mean time to resolution.
• Share Deployment health visibility across teams.

​

Additional improvements

• Now, when you link directly to your Astro or Airflow UI, any link previews successfully have improved visuals and include metadata information.
• Improved the Astro Alerts UI to streamline creating notification channels and alerts across dags and Deployments. See Astro Alerts for setup information.
• Added an example Universal Metrics Export dashboard configuration file for Grafana Cloud. See Export metrics from Astro for setup instructions and Grafana example for a configuration example.
• Organization Dashboards are now generally available to Enterprise-tier customers. See View Organization Dashboards for more information.

​

Bug fixes

• Fixed a bug where the Astro UI would show an error in your Deploy History instead of your Runtime Version, if the Runtime version is yanked.
• Fixed an issue when creating an alert where you couldn’t select tasks with the same task_id across multiple selected dags.

​

Run your first dag on Astro with GitHub Integration

​

Work with dbt projects on Astro

​

Additional improvements

• Improved how the GitHub Integration in the Astro UI links directly to the Astro Project in a GitHub repository.
• (Astro Hosted only) The customer managed workload identity setting for Deployments is now generally available. This allows you to grant Astro Deployments all of the permissions of an AWS IAM role. See Attach an IAM role to your Deployment for detailed information.
• The ability to deploy automatically from GitHub using the official Astro GitHub integration is now generally available. See Deploy code with GitHub for setup steps.
• Astro Runtime version 11 is now classified as Long Term Support (LTS). This means that instead of being supported until October 2024, Astro will support Runtime version 11 until October 2025. See the Astro Runtime maintenance and lifecycle schedule for more information, including information about restricted versions.

​

Bug fixes

• Fixed some bugs in the Astro UI for defining ephemeral storage to show storage limits, fix maximum pod size for storage, and add storage for default pod size.

​

Additional improvements

• Added the following metrics to Metrics Export:
  • The new airflow_executor_open_slots, airflow_dagrun_dependency-check, and airflow_dagrun_dependency-check.<dag_id> metrics allow you to collect metrics about your dags and executor status.
  • Monitor task execution with kube_pod_container_resource_limits. This new metric enables you to track resource use against the configured limits so you can understand if task execution meets your configured CPU, memory, or storage limits for your Celery Workers or Kubernetes Executor and KubernetesPodOperator pods.
• Added new supporting documentation for the Astro Terraform Provider, including a Getting Started guide and code examples for common uses. Read Astro Terraform Provider for more information.

​

Bug fixes

• Fix an error where start times in the UI were different from actual dag run trigger times.

​

Additional improvements

• Fixed a bug that prevented adding a date input for a dag filter in the Astro UI.

​

Export metrics about your Astro Deployments to observability tools

​

Added ephemeral storage metrics to Astro Deployment Analytics

• Ephemeral Storage Usage metric that shows a % of usage against the configured limit for your Celery Workers, KubernetesPodOperator, Kubernetes Executor, and Scheduler.
• Dynamic Y-axis Scaling to the Celery Worker, KPO/KE, and Scheduler Deployment analytics panel to include dynamic zooming.

​

Additional improvements

• The Scheduler heartbeat not found Deployment health incident is downgraded to Warning from Critical.

​

Customer managed workload identity for AWS

• Re-use or share a customer managed identity across many Deployments, either ephemeral or static.
• Leverage existing identities when migrating from MWAA or open source Airflow environments

​

Self-healing workers automatically address stuck queued tasks

• It first identifies workers that both have tasks stuck in a queued state for an extensive time period and are in Deployments where the concurrency available means that tasks should not be queued.
• The healer then kills catatonic workers that are not running tasks or shifts workers that are still running tasks into a warm shutdown period.
• After the catatonic worker is shut down, a new healthy, worker comes online and resumes tasks.

​

Additional Improvements

• Improved the formatting for how IP addresses are listed in the Astro UI to make it easier to copy and paste them.

​

Bug fixes

• europe-west6 is no longer available as a region for dedicated clusters on GCP.

​

Additional Improvements

• Moved the Environment variable tab in Deployment Settings to the Environment tab, and renamed it to Environment Variables. This helps disambiguate between Environment Variables and Airflow Variables in the Astro UI. Refer to Manage environment variables for more information about the different methods to set up and manage Environment variables.

​

Bug fixes

• Fixed a bug where a duplicate Customer managed identity option displayed when configuring Deployment settings.

​

Restrict Astro access to specific IP Address ranges with IP Access list

​

Additional improvements

• When using the GitHub integration, you can now retry a failed deploy or trigger a Git deploy directly from your Deployment settings page in the Astro UI. See GitHub Integration for more information.
• Deployments at the end of an Astro trial now automatically hibernate and are not immediately deleted. If your trial has ended, add a credit card number to your Organization to access your Workspaces and wake your hibernating Deployment. While in hibernation, Deployment settings and configurations are preserved. If you don’t enter a payment method within 30 days, your hibernating Deployments and all corresponding data are deleted. See Start a trial for more information.
• The ability to create, update, and delete Deployment API tokens is restricted to users with the Workspace Owner role. Read Workspace user permissions for more information.

​

Bug fixes

• Fixed an issue where the Organization Settings Dashboard failed to load correctly.

​

API keys are no longer supported

​

Configure ephemeral storage on worker Pods

• Celery worker: 10 GiB minimum by default. 100 GiB maximum.
• Kubernetes executor/ Kubernetes pod operator: 0.25 GiB minimum by default. 100 GiB maximum.

​

Automate Airflow, resource, and infrastucture management with the Astro Terraform Provider

​

Additional improvements

• The Astro UI now validates cron expressions for development Deployment hibernation schedules and shows commonly used hibernation schedules that you can enable or disable. Refer to Create a hibernation schedule for more information.
• If you don’t already have a GitHub repository, you can now create one when you authorize the GitHub Integration from the Astro UI. Previously, you could only connect Astro to existing repositories.

​

Bug fixes

• Fixed an issue where you could not use Airflow connection testing with Azure Managed Identities on Astro.

​

Additional improvements

• Dedicated clusters are now available only to Team tier customers and above.
• Added a centralized docs reference page that lists all open source Apache Airflow provider packages and their versions for each Astro Runtime version. See Provider package reference.

​

Bug fixes

• Fixed an issue where you could make changes to clusters and Deployments on an inactive Organization.
• The default storage resources for the Kubernetes executor and KubernetesPodOperator Pods are now enforced with 0.25Gi instead of 10Gi in some cases. You can still customize the resource allocation for Kubernetes pods depending on your needs. See Configure Kubernetes Pod Resources

​

The Astro GitHub integration is now in Preview

​

Bug fixes

• Fixed an issue where users with custom roles could see roles that could not be assigned in the Astro UI.

​

Additional improvements

• Airflow connections that you configure through the Astro UI environment manager are now mounted to Deployment schedulers, meaning that scheduler processes can now make use of these Airflow connections.

​

Bug fixes

• Fixed an issue where you couldn’t update users who were added through SCIM but didn’t belong to an Organization.
• Fixed an issue where a user with a custom role could create API Tokens, users, or teams with greater permissions than their own.

​

Updates to address ranges for dedicated clusters on Google Cloud Provider

​

Improvements to Astro performance

​

Additional improvements

• You can no longer create Deployments using Astro Runtime versions marked as yanked in https://updates.astronomer.io/astronomer-runtime, even if your Organization has enabled creating Deployments with deprecated Runtime versions. These versions of the Astro Runtime have known issues and should not be used. For more information, see Restricted Runtime Versions.

​

Bug fixes

• Fixed a bug where multiple users could not access Organization Dashboards simultaneously.

​

Deploy automatically from GitHub using the official Astro GitHub integration

• Allows you to enforce software development best practices without maintaining custom CI/CD scripts.
• Enables developers to iterate on dag code quickly.
• Shows Git metadata directly in the Astro UI, including Git commit descriptions.
• Gives you greater visibility into the status and detailed logs of an individual deploy.

​

Restrict a custom Deployment role to specific Workspaces

​

Additional improvements

• The custom Deployment roles feature is now generally available.
• You can now promote a development Deployment to a production Deployment by switching off the Development Mode toggle in the Deployment’s configuration.
• Workspace Members can now see and use custom Airflow menu items. To give a custom role this permission, you can add deployment.airflow.customMenu.get to the role’s permissions list. This permission works only on Deployments running Astro Runtime 9 or later. Note that you might have to modify the code for your menu item plugins to make them work on Astro. See Appbuilder menu items for more information.
• You can now filter the Workspaces and clusters lists in the Astro UI by name.

​

Bug fixes

• To improve the reliability of data lineage for customers who leverage it, data lineage is now a private preview feature that can be enabled upon request. To reenable data lineage for an Astro Organization, reach out to your account team.

​

Bug fixes

• Fixed an issue where you couldn’t grant a custom Deployment role to a Deployment API token using the Astro API.

​

Additional improvements

• You can now manage custom Deployment roles using the Astro API.
• Improved the time it takes to load Deployment analytics in the Astro UI.
• You can now view info-level incidents from the Deployment health status indicator in the Astro UI.
• It is now possible for workers to use up to 6400 CPUs and 12800 GiB of memory on a single Deployment.

​

Bug fixes

• Fixed an issue where you couldn’t configure boolean values for Airflow connections in the Astro UI.
• Fixed an issue where Airflow connections configured through the Astro UI did not work with deferrable tasks.

​

Additional improvements

• Deployment health incidents are now generally available.
• Deployment hibernation is now in public preview.

​

Bug fixes

• Fixed an issue where connections configured in the Astro UI were not mounted to triggerer Pods, resulting in failed runs for tasks that use deferrable operators.
• Fixed an issue where a field in the Snowflake - Private Key (Content) connection type was not parsed correctly.
• Fixed an issue where Organization Members could delete and update Astro alerts.
• Fixed an issue where the Astro UI didn’t show the correct amount of available ephemeral storage for the default worker queue.
• Removed additional dependencies to make Astro more resilient to Quay outages.

​

Refactored Astro API documentation

• Format and test API requests directly in your browser.
• Export requests to Python, Javascript, and curl.
• View weekly changelogs for the API.

​

Create Deployments with deprecated versions of Astro Runtime

​

New GCP database instance types available

• XLarge Compute Optimized (24 CPU, 48 GiB MEM)
• XXLarge Compute Optimized (32 CPU, 64 GiB MEM)

​

Additional improvements

• The Astro UI now includes Learning Bytes for features that are not yet configured within your Organization.
• The Astro UI now loads the status for dag and task runs more quickly.

​

Bug fixes

• When you change a worker type for an existing worker queue, the Astro UI no longer resets the worker queue’s concurrency configurations.
• Fixed an issue where the Astro API did not return the correct value for IsHibernating when you queried Deployment information.

​

New Azure regions available on Astro Hosted

• centralus
• westus3
• southcentralus

​

Custom Deployment roles are now in Preview

​

Export data from reporting dashboards using webhooks

​

Additional improvements

• You can now use the new Credits tab in the Organization Billing page to see your credit balance.
• On Astro Hybrid, the maximum worker concurrency on a worker queue has increased from 64 to 256.

​

Reporting dashboards are now in public preview

​

Customize Deployment-level permissions using Deployment roles

​

New Deployment registry cache to improve resiliency

​

Additional improvements

• Removed nonfunctional network usage per Pod metrics from the Deployment Analytics page.
• The end-of-life date for Deployment API keys is June 1, 2024.
• The Cloud UI has been renamed to the Astro UI across all help text and documentation.
• Due to a minor change to Astronomer cluster architecture, you can no longer add custom tags to Hybrid clusters on AWS.

​

Bug fixes

• Fixed an issue where CPU usage per Pod metrics did not render correctly in the Deployment Analytics page.
• Fixed an issue where the Astro UI didn’t show all available Teams when selecting Teams to add to a Workspace.

​

Use a custom service account to authorize Deployments to GCP

​

Bug fixes

• Fixed an issue where the Astro API failed to list Deployments after you deleted a hibernation override setting.

​

New worker types

​

New platform variables to improve Celery executor reliability

AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT=5
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE=True
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT=True

​

Ephemeral storage limit on schedulers

• Your Deployment is not producing large amounts of temporary files without cleaning them up.
• If you’re using the BingAds Python SDK, your Deployment uses version 13.0.14 or later. Earlier releases include a bug that generates large amounts of temporary files.

​

Bug fixes

• You can no longer create Teams using the Astro API in an Organization that has SCIM provisioning enabled.
• Astro API responses have been standardized to always return cloud provider names in uppercase (for example, AWS).

​

New Astro reporting dashboards show metrics for Deployments across your Organization

​

Additional improvements

• When you submit a support request from the Astro UI, you must now define an Active Engagement Period when you or a member of your team can engage with a member of Astronomer support.
• Workspace Members can now access the Clusters view in the Airflow UI for a Deployment.

​

Bug fixes

• Fixed an issue where network connections between clusters could be disrupted occasionally.
• When you retrieve information about a Deployment through the Astro API, the API now returns an empty value for EnvironmentVariables if the Deployment has no environment variables.
• Deleting a Workspace through the Astro API now deletes all Astro Cloud IDE projects associated with the Workspace.
• Fixed an issue where you could not clear optional fields in a Deployment’s configuration using the Astro API.

​

Bug fixes

• Fixed an issue where all Deployment task logs included the error Not exporting configs to configmap...

​

New messages for Deployment health status

​

Additional improvements

• You can now access Ask Astro from the Astro UI Help menu:
  
• User role titles are now consistently formatted across the Astro UI.

​

Self-service VPC peering and route management for AWS

​

Additional improvements

• There is no longer a minimum on the amount of CPU and memory that you can request for Kubernetes Pods.

​

Bug fixes

• Fixed an issue in Astro Hybrid where refreshing the browser could occasionally reset a worker queue’s worker type setting.

​

Additional improvements

• The Astro UI Deployment analytics page now shows CPU Usage Per Pod (%) and Memory Usage Per Pod (MB) as a percentage of your total available resources rather than the resources of a single worker Pod, such that these metrics will never show Deployment resources usage as exceeding 100%.
• The maximum value for worker queue Max # of workers has increased from 30 to 100.

​

Additional improvements

• When you create a Deployment, the Astro UI now shows the Airflow version of your Deployment instead of the equivalent Astro Runtime version.
• Enabled point-in-time restore (PITR) on Hybrid GCP clusters to improve resiliency to outages. Note that this might result in increased costs for cloud storage.

​

Bug fixes

• Fixed an issue where you occasionally couldn’t access the Airflow UI for a Deployment with the error “No healthy upstream”.
• Fixed an issue where deleting a user with no Workspace membership from an Organization would affect how their Workspace membership appeared in other, unrelated Organizations.
• Fixed an issue where the Open in Airflow button on the dag details page in the Astro UI did not open the Airflow UI as expected.