AWS Hybrid cluster settings
This document applies only to Astro Hybrid. To see whether you’re an Astro Hybrid user, click Organization Settings. Your Astro product type is listed under Product Type on the General page.
To create a cluster on Astro Hosted, see Create a dedicated cluster.
Unless otherwise specified, new clusters on Astro are created with a set of default AWS resources that should be suitable for most use cases.
Read the following document for a reference of our default resources as well as supported cluster configurations.
Default cluster values
Supported cluster regions
You can host Astro Hybrid clusters in the following AWS regions:
Modifying the region of an existing cluster on Astro is not supported. If you’re interested in an AWS region that isn’t listed, contact Astronomer support.
Supported RDS instance types
The following AWS RDS instance types are supported on Astro:
db.m6g
- db.m6g.large (default)
- db.m6g.xlarge
- db.m6g.2xlarge
- db.m6g.4xlarge
- db.m6g.8xlarge
- db.m6g.12xlarge
- db.m6g.16xlarge
db.r6g
- db.m6g.large (default)
- db.m6g.xlarge
- db.m6g.2xlarge
- db.m6g.4xlarge
- db.m6g.8xlarge
- db.m6g.12xlarge
- db.m6g.16xlarge
db.r5
- db.r5.large
- db.r5.xlarge
- db.r5.2xlarge
- db.r5.4xlarge
- db.r5.8xlarge
- db.r5.12xlarge
- db.r5.16xlarge
- db.r5.24xlarge
db.m5
- db.m5.large
- db.m5.xlarge
- db.m5.2xlarge
- db.m5.4xlarge
- db.m5.8xlarge
- db.m5.12xlarge
- db.m5.16xlarge
- db.m5.24xlarge
For detailed information about each instance type, see Amazon RDS Instance Types. If you’re interested in an RDS instance type that is not on this list, contact Astronomer support.
Supported worker node pool instance types
Each worker in a worker node pool runs a single worker Pod. A worker Pod’s actual available size is equivalent to the total capacity of the instance type minus Astro’s system overhead.
The following table lists all available instance types for worker node pools, as well as the Pod size that is supported for each instance type. As the system requirements of Astro change, these values can increase or decrease.
If your Organization is interested in using an instance type that supports a larger worker size, contact Astronomer support. For more information about configuring worker size on Astro, see Deployment resources.
t series instance types in standard mode for production workloads, because CPU utilization for t instance types in standard mode can be throttled.Ephemeral storage
With the exception of m5d and m6id nodes, all supported node types have a maximum of 20GiB of storage per node for system use only. If you need locally attached storage for task execution, Astronomer recommends modifying your cluster to run m5d or m6id nodes, which AWS provisions with NVMe SSD volumes. To utilize the ephemeral storage on these node types, have your task write data to /ephemeral. If your task uses the KubernetesPodOperator, mount an emptyDir volume in your operator container spec instead. See Amazon EC2 M6i Instances and Amazon EC2 M5 Instances for the amount of available storage in each node type.
If you need to pass significant data between Airflow tasks, Astronomer recommends using an XCom backend such as AWS S3 or Google Cloud Storage (GCS). For more information and best practices, see the Airflow Guide on Passing Data Between Airflow Tasks.