Skip to main content

AWS Hybrid cluster settings

warning

This document applies only to Astro Hybrid. To see whether you're an Astro Hybrid user, click Organization Settings. Your Astro product type is listed under Product Type on the General page.

To create a cluster on Astro Hosted, see Create a dedicated cluster.

Unless otherwise specified, new clusters on Astro are created with a set of default AWS resources that should be suitable for most use cases.

Read the following document for a reference of our default resources as well as supported cluster configurations.

Default cluster values

ResourceDescriptionQuantity/ Default SizeConfigurable
EKS ClusterAn EKS cluster is required to run the Astro data plane, which hosts the resources and data required to execute Airflow tasks.1x
Worker node poolA node pool of EC2 instances that hosts all workers with the default worker type for all Deployments in the cluster. The number of nodes in the pool auto-scales based on the demand for workers in your cluster. You can configure additional worker node pools to run tasks on different worker types.1x pool of m5.xlarge nodesYes. See Manage worker node pools.
Airflow node poolA node pool of EC2 instances that runs all core Airflow components, including the scheduler and webserver, for all Deployments in the cluster. This node pool is fully managed by Astronomer.1x pool of m5.xlarge nodes
Astro system node poolA node pool of EC2 instances that runs all other system components required in Astro. The availability zone determines how many nodes are created. This node pool is fully managed by Astronomer.1x pool of m5.xlarge nodes
RDS for PostgreSQL InstanceThe RDS instance is the primary database of the Astro data plane. It hosts a metadata database for each Deployment in the cluster. All RDS instances on Astro are multi-AZ.1x db.m6g.largeYes. See Configure your relational database.
Elastic IPsRequired for connectivity with the Astro control plane and other public services.2x
SubnetsSubnets are provisioned in 2 different Availability Zones (AZs) for redundancy, with 1 public and 1 private subnet per AZ. Public subnets are required for the NAT and Internet gateways, while private subnets are required for EC2 nodes.2x /26 (public) and 1x /21 + 1x /22 (private)Yes. See Connect Astro to AWS data sources.
Internet GatewayRequired for connectivity with the control plane and other public services.1x
NAT GatewaysNAT Gateways translate outbound traffic from private subnets to public subnets.2x
RoutesRoutes are necessary to direct network traffic from the subnets and gateways.2x
Route TablesHome for the routes.2x
VPCVirtual network for launching and hosting AWS resources.1x /20Yes. See Connect Astro to AWS data sources.
VPC Interface Endpoint ServicesEnables AWS PrivateLink connectivity to Elastic Load Balancing (ELB), AWS Auto Scaling plans, AWS Security Token Service (AWS STS), and Amazon Elastic Container Registry (ECR) Docker and API services.5x
Amazon S3Stores Airflow task logs.1x
S3 Gateway EndpointIP routes for Amazon Simple Storage Service (Amazon S3)1x

Supported cluster regions

You can host Astro Hybrid clusters in the following AWS regions:

CodeName
af-south-1Africa (Cape Town)
ap-east-1Asia Pacific (Hong Kong)
ap-northeast-1Asia Pacific (Tokyo)
ap-northeast-2Asia Pacific (Seoul)
ap-northeast-3Asia Pacific (Osaka)
ap-southeast-1Asia Pacific (Singapore)
ap-southeast-2Asia Pacific (Sydney)
ap-south-1Asia Pacific (Mumbai)
ca-central-1Canada (Central)
eu-central-1Europe (Frankfurt)
eu-south-1Europe (Milan)
eu-west-1Europe (Ireland)
eu-west-2Europe (London)
eu-west-3Europe (Paris)
me-south-1Middle East (Bahrain)
sa-east-1South America (São Paulo)
us-east-1US East (N. Virginia)
us-east-2US East (Ohio)
us-west-1US West (N. California)
us-west-2US West (Oregon)

Modifying the region of an existing cluster on Astro is not supported. If you're interested in an AWS region that isn't listed, contact Astronomer support.

Supported RDS instance types

The following AWS RDS instance types are supported on Astro:

db.m6g

  • db.m6g.large (default)
  • db.m6g.xlarge
  • db.m6g.2xlarge
  • db.m6g.4xlarge
  • db.m6g.8xlarge
  • db.m6g.12xlarge
  • db.m6g.16xlarge

db.r6g

  • db.m6g.large (default)
  • db.m6g.xlarge
  • db.m6g.2xlarge
  • db.m6g.4xlarge
  • db.m6g.8xlarge
  • db.m6g.12xlarge
  • db.m6g.16xlarge

db.r5

  • db.r5.large
  • db.r5.xlarge
  • db.r5.2xlarge
  • db.r5.4xlarge
  • db.r5.8xlarge
  • db.r5.12xlarge
  • db.r5.16xlarge
  • db.r5.24xlarge

db.m5

  • db.m5.large
  • db.m5.xlarge
  • db.m5.2xlarge
  • db.m5.4xlarge
  • db.m5.8xlarge
  • db.m5.12xlarge
  • db.m5.16xlarge
  • db.m5.24xlarge

For detailed information about each instance type, see Amazon RDS Instance Types. If you're interested in an RDS instance type that is not on this list, contact Astronomer support.

Supported worker node pool instance types

Each worker in a worker node pool runs a single worker Pod. A worker Pod's actual available size is equivalent to the total capacity of the instance type minus Astro’s system overhead.

The following table lists all available instance types for worker node pools, as well as the Pod size that is supported for each instance type. As the system requirements of Astro change, these values can increase or decrease.

Worker Node TypeCPUMemory
m5.xlarge2 CPUs14.5 GiB MEM
m5.2xlarge6 CPUs30.5 GiB MEM
m5.4xlarge14 CPUs62.5 GiB MEM
m5.8xlarge30 CPUs126.5 GiB MEM
m5.12xlarge46 CPUs190.5 GiB MEM
m5.16xlarge62 CPUs254.5 GiB MEM
m5.24xlarge94 CPUs382.5 GiB MEM
m5.metal94 CPUs382.5 GiB MEM
m5d.xlarge2 CPUs14.5 GiB MEM
m5d.2xlarge6 CPUs30.5 GiB MEM
m5d.4xlarge14 CPUs62.5 GiB MEM
m5d.8xlarge30 CPUs126.5 GiB MEM
m5d.12xlarge46 CPUs190.5 GiB MEM
m5d.16xlarge62 CPUs254.5 GiB MEM
m5d.24xlarge94 CPUs382.5 GiB MEM
m5d.metal94 CPUs382.5 GiB MEM
m6i.xlarge2 CPUs14.5 GiB MEM
m6i.2xlarge6 CPUs30.5 GiB MEM
m6i.4xlarge14 CPUs62.5 GiB MEM
m6i.8xlarge30 CPUs126.5 GiB MEM
m6i.12xlarge46 CPUs190.5 GiB MEM
m6i.16xlarge62 CPUs254.5 GiB MEM
m6i.24xlarge94 CPUs382.5 GiB MEM
m6i.metal94 CPUs382.5 GiB MEM
m6id.xlarge2 CPUs14.5 GiB MEM
m6id.2xlarge6 CPUs30.5 GiB MEM
m6id.4xlarge14 CPU62.5 GiB MEM
m6id.8xlarge30 CPU126.5 GiB MEM
m6id.12xlarge46 CPU190.5 GiB MEM
m6id.16xlarge62 CPU254.5 GiB MEM
m6id.24xlarge94 CPU382.5 GiB MEM
m6id.metal126 CPU510.5 GiB MEM
r6i.xlarge2 CPUs30.5 GiB MEM
r6i.2xlarge6 CPUs62.5 GiB MEM
r6i.4xlarge14 CPUs126.5 GiB MEM
r6i.8xlarge30 CPUs254.5 GiB MEM
r6i.12xlarge46 CPUs382.5 GiB MEM
r6i.16xlarge62 CPUs510.5 GiB MEM
r6i.24xlarge94 CPUs766.5 GiB MEM
r6i.metal94 CPUs1022.5 GiB MEM
c6i.xlarge2 CPUs6.5 GiB MEM
c6i.2xlarge6 CPUs14.5 GiB MEM
c6i.4xlarge14 CPUs30.5 GiB MEM
c6i.8xlarge30 CPUs62.5 GiB MEM
c6i.12xlarge46 CPUs94.5 GiB MEM
c6i.16xlarge62 CPUs126.5 GiB MEM
c6i.24xlarge94 CPUs190.5 GiB MEM
c6i.metal126 CPUs254.5 GiB MEM
t2.xlarge2 CPUs14.5 GiB MEM
t3.xlarge2 CPUs14.5 GiB MEM
t3.2xlarge6 CPUs30.5 GiB MEM

If your Organization is interested in using an instance type that supports a larger worker size, contact Astronomer support. For more information about configuring worker size on Astro, see Deployment resources.

warning

Astronomer doesn’t recommend using t series instance types in standard mode for production workloads, because CPU utilization for t instance types in standard mode can be throttled.

Ephemeral storage

With the exception of m5d and m6id nodes, all supported node types have a maximum of 20GiB of storage per node for system use only. If you need locally attached storage for task execution, Astronomer recommends modifying your cluster to run m5d or m6id nodes, which AWS provisions with NVMe SSD volumes. To utilize the ephemeral storage on these node types, have your task write data to /ephemeral. If your task uses the KubernetesPodOperator, mount an emptyDir volume in your operator container spec instead. See Amazon EC2 M6i Instances and Amazon EC2 M5 Instances for the amount of available storage in each node type.

If you need to pass significant data between Airflow tasks, Astronomer recommends using an XCom backend such as AWS S3 or Google Cloud Storage (GCS). For more information and best practices, see the Airflow Guide on Passing Data Between Airflow Tasks.

Was this page helpful?