Create a dedicated Astro cluster
A dedicated cluster exclusively runs Deployments from your Organization within a single-tenant environment on Astronomer's cloud. Dedicated clusters provide more configuration options for regions, connectivity, and security than standard clusters. You might want to create a dedicated cluster if:
- You need to connect to data services using private networking.
- You want more options for the region your cluster is hosted in.
- You otherwise want to keep your Deployments as isolated as possible.
Dedicated clusters offer the self-service convenience of a fully managed service while respecting the need to keep data private, secure, and within a single-tenant environment. If you don't need the aforementioned features, you can use one of the standard clusters when you Create a Deployment.
Setup
- AWS
- GCP
- Azure
-
In the Astro UI, click Organization Settings, then click Clusters.
-
Click + Cluster.
-
Configure the following details about your cluster:
- Cloud Provider: Select AWS.
- Name: The name for your cluster.
- Region: Select the region that you want your cluster to run in.
- VPC Subnet Range: Provide a subnet range for Astro to connect to your existing AWS resources through VPC peering. The default is
172.20.0.0/20, and the minimum size you can use is/21.
-
Click Create cluster. After Astro finishes creating the cluster, users in your Organization can select the cluster when they create a Deployment.
-
In the Astro UI, click Organization Settings, then click Clusters.
-
Click + Cluster.
-
Configure the following details about your cluster:
- Cloud Provider: Select GCP.
- Name: The name for your cluster.
- Region: Select the region that your cluster runs in.
- VPC Subnet Range: Specify the range used by nodes in your GKE cluster (Default:
172.20.0.0/22). Astro uses this range to make private connections to your target data sources.
-
(Optional) Configure the following Advanced Configuration details about your cluster for Private Networking.
Astro uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources and defaults secondary ranges to RFC 6598 address space (non-standard Private IP addresses), to minimize the risk and concern with IP overlap and exhaustion. When using private networking, like VPN or VPC Peering, between Astro and your target data sources, your target data sources see connections from the default VPC Subnet Range. If you're using private connections, confirm that the following Advanced Configuration network ranges do not overlap with your target data source networks.
- Pod Subnet Range: Specify the range used by GKE Pods (Default:
100.64.0.0/16). - Service Subnet Range: Specify the range used by Services in your GKE cluster (Default:
100.65.0.0/22). - Service Peering Range: Specify the range used by Private Service connections (Default:
100.66.0.0/21)
If there is an overlap between the Advanced Configurations and your target data source networks, you can use the following alternative ranges:
- RFC 1918:
- 10.0.0.0/8, 10.0.0.0 – 10.255.255.255
- 172.16.0.0/12, 172.16.0.0 – 172.31.255.255
- 192.168.0.0/16, 192.168.0.0 – 192.168.255.255
- RFC 6598: 100.64.0.0/10, specifically IP addresses from 100.64.0.0 to 100.127.255.255
- Pod Subnet Range: Specify the range used by GKE Pods (Default:
-
Click Create cluster. After Astro finishes creating the cluster, users in your Organization can select the cluster when they create a Deployment.
All GCP dedicated clusters are subscribed to the GKE regular release channel, meaning that Google automatically upgrades the cluster and its nodes whenever an upgrade is available.
After you create a GCP cluster, you can control when these upgrades happen by requesting a maintenance window for the cluster. Maintenance windows determine when and how Google updates your cluster. You can use maintenance windows to ensure that upgrades don't happen while critical DAGs are running on your cluster.
To set a maintenance window, first choose a maintenance window time and read through the maintenance window considerations to make sure that the time is optimized for your cluster. Then, contact Astronomer Support and provide your cluster ID and desired maintenance window.
-
In the Organization section of the Astro UI, click Organization Settings, then click Clusters, then select a cluster.
-
Click + Cluster.
-
Configure the following details about your cluster:
- Cloud Provider: Select Azure.
- Name: The name for your cluster.
- Region: Select the region that you want your cluster to run in.
- VPC Subnet Range: Provide a subnet range for Astro to connect to your existing Azure resources through a VNet connection. The default is
172.20.0.0/19.
-
Click Create cluster. After Astro finishes creating the cluster, users in your Organization can select the cluster when they create a Deployment.