Astro HybridResource reference

GCP Hybrid cluster settings

This document applies only to Astro Hybrid. To see whether you’re an Astro Hybrid user, click Organization Settings. Your Astro product type is listed under Product Type on the General page.

To create a cluster on Astro Hosted, see Create a dedicated cluster.

Unless otherwise specified, new Clusters on Google Cloud Platform (GCP) are created with a set of default resources that our team has deemed appropriate for most use cases.

Read the following document for a reference of our default resources as well as supported cluster configurations.

Default cluster values

ResourceDescriptionQuantity/ Default SizeConfigurable
GKE ClusterA GKE cluster is required to run the Astro data plane, which hosts the resources and data required to execute Airflow tasks. Workload Identity is enabled on this cluster.1x, IP Ranges are 172.21.0.0/19 for cluster IPs and 172.22.0.0/22 for cluster services
Worker node poolA node pool that hosts all workers with the default worker type for all Deployments in the cluster. The number of nodes in the pool auto-scales based on the demand for workers in your cluster. You can configure additional worker node pools to run tasks on different worker types.1x pool of e2-standard-4 nodesYes. See Manage worker node pools.
Airflow node poolA node pool that runs all core Airflow components, including the scheduler and webserver, for all Deployments in the cluster. This node pool is fully managed by Astronomer.1x pool of e2-standard-4 nodes
Astro system node poolA node pool that runs all other system components required in Astro. The availability zone determines how many nodes are created. This node pool is fully managed by Astronomer.1x pool of e2-standard-4 nodes
Cloud SQL for PostgreSQLThe Cloud SQL instance is the primary database for the Astro data plane. It hosts the metadata database for each Airflow Deployment hosted on the GKE cluster. All Cloud SQL instances are multi-AZ.1x regional instance with 2 vCPUs, 8GiB memoryYes. See Configure your relational database.
VPCVirtual private network for hosting GCP resources1x /19Yes. See Connect Astro to GCP data sources.
SubnetA single subnet is provisioned in the VPC.1, IP Range is 172.20.0.0/22
Service Network PeeringThe Astro VPC is peered to the Google Service Networking VPC.1, IP Range is 172.23.0.0/20
NAT Router (External)Required for connectivity with the Astro control plane and other public services1.
Workload Identity PoolAstro uses the fixed Workload Identity Pool for your cluster. One is created if it does not exist.PROJECT_ID.svc.id.goog
Google Cloud Storage (GCS) BucketStores Airflow task logs.1 bucket with name airflow-logs-<clusterid>

Supported cluster regions

You can host Astro Hybrid clusters in the following GCP regions:

CodeName
asia-east1Taiwan, Asia
asia-northeast1Tokyo, Asia
asia-northeast2Osaka, Asia
asia-northeast3Seoul, Asia
asia-south1Mumbai, Asia
asia-south2Delhi, Asia
asia-southeast1Singapore, Asia
asia-southeast2Jakarta, Asia
australia-southeast1Sydney, Australia
australia-southeast2Melbourne, Australia
europe-north1Finland, Europe
europe-southwest1Madrid, Europe
europe-west1Belgium, Europe
europe-west2England, Europe
europe-west3Frankfurt, Europe
europe-west4Netherlands, Europe
europe-west6Zurich, Europe
europe-west8Milan, Europe
europe-west9Paris, Europe
northamerica-northeast1Montreal, North America
southamerica-east1Sau Paolo, South America
us-central1Iowa, North America
us-east1South Carolina, North America
us-east4Virginia, North America
us-east5Columbus, North America
us-west1Oregon, North America
us-west2Los Angeles, North America
us-west4Nevada, North America

Modifying the region of an existing Astro cluster isn’t supported. If you’re interested in a GCP region that isn’t listed, contact Astronomer support.

Supported Cloud SQL instance types

The following Cloud SQL instance types are supported on Astro:

  • Small General Purpose (2 CPU, 8 GiB MEM)
  • Medium General Purpose (4 CPU, 16 GiB MEM)
  • Large General Purpose (8 CPU, 32 GiB MEM)
  • Small Memory Optimized (2 CPU, 12 GiB MEM)
  • Medium Memory Optimized (4 CPU, 24 GiB MEM)
  • Large Memory Optimized (8 CPU, 36 GiB MEM)
  • Small Compute Optimized (4 CPU, 8 GiB MEM)
  • Medium Compute Optimized (8 CPU, 16 GiB MEM)
  • Large Compute Optimized (16 CPU, 32 GiB MEM)
  • XLarge Compute Optimized (24 CPU, 48 GiB MEM)
  • XXLarge Compute Optimized (32 CPU, 64 GiB MEM)

For detailed information about each instance type, see the Cloud SQL documentation. If you’re interested in an Cloud SQL instance type that is not on this list, contact Astronomer support.

Supported worker node pool instance types

Each worker node in a pool runs a single worker Pod. A worker Pod’s actual available size is equivalent to the total capacity of the instance type minus Astro’s system overhead.

The following table lists all available instance types for worker node pools, as well as the Pod size that is supported for each instance type. As the system requirements of Astro change, these values can increase or decrease.

Node Instance TypeCPUMemory
e2-standard-42 CPUs14.5 GiB MEM
e2-standard-86 CPUs30.5 GiB MEM
e2-standard-1614 CPUs62.5 GiB MEM
e2-standard-3230 CPUs126.5 GiB MEM
e2-highmem-42 CPUs30.5 GiB MEM
e2-highmem-86 CPUs62.5 GiB MEM
e2-highmem-1614 CPUs126.5 GiB MEM
e2-highcpu-42 CPUs2.5 GiB MEM
e2-highcpu-86 CPUs6.5 GiB MEM
e2-highcpu-1614 CPUs14.5 GiB MEM
e2-highcpu-3230 CPUs30.5 GiB MEM
n2-standard-42 CPUs14.5 GiB MEM
n2-standard-86 CPUs30.5 GiB MEM
n2-standard-1614 CPUs62.5 GiB MEM
n2-standard-3230 CPUs126.5 GiB MEM
n2-standard-4840 CPUs190.5 GiB MEM
n2-standard-6462 CPUs254.5 GiB MEM
n2-highmem-42 CPUs30.5 GiB MEM
n2-highmem-86 CPUs60.5 GiB MEM
n2-highmem-1614 CPUs126.5 GiB MEM
n2-highmem-3230 CPUs254.5 GiB MEM
n2-highmem-4848 CPUs382.5 GiB MEM
n2-highmem-6462 CPUs510.5 GiB MEM
n2-highcpu-42 CPUs2.5 GiB MEM
n2-highcpu-86 CPUs6.5 GiB MEM
n2-highcpu-1614 CPUs14.5 GiB MEM
n2-highcpu-3230 CPUs30.5 GiB MEM
n2-highcpu-4846 CPUs46.5 GiB MEM
n2-highcpu-6462 CPUs62.5 GiB MEM
c2-standard-42 CPUs14.5 GiB MEM
c2-standard-86 CPUs30.5 GiB MEM

If your Organization is interested in using an instance type that supports a larger worker size, contact Astronomer support. For more information about configuring worker size on Astro, see Deployment settings.