Version:

v0.10.0

Documentation

Configuring Resources with Helm


Default Resources

By default, Astronomer needs around 10 CPUs and 44Gi of memory:

PodRequest CPURequest MemLimit CPULimit MemStorage
orbit100m256Mi500m1024MiNA
houston250m512Mi800m1024MiNA
prisma250m512Mi500m1024MiNA
commander250m512Mi500m1024MiNA
registry250m512Mi500m1024Mi100Gi
install100m256Mi500m1024MiNA
nginx500m1024Mi12048MiNA
grafana250m512Mi500m1024MiNA
prometheus1000m4Gi1000m4Gi100Gi
elasticsearch client replica-112Gi24GiNA
elasticsearch client replica-212Gi24GiNA
elasticsearch data replica-112Gi24Gi100Gi
elasticsearch data replica-212Gi24Gi100Gi
elasticsearch master replica-112Gi24Gi20Gi
elasticsearch master replica-212Gi24Gi20Gi
elasticsearch master replica-312Gi24Gi20Gi
kibana250m512Mi500m1024MiNA
fluentd250m512Mi500m1024MiNA
kubeState250m512Mi500m1024MiNA
Total10.723.5Gi21.344Gi460Gi

Changing Values

You can change the request and limit of any of the components above in your config.yaml or in values.yaml (config.yaml will overwrite values.yaml).

To change something like the resources allocated to Orbit, add

#####
#Changing Orbit CPU
####

astronomer:
  orbit:
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "700m"
        memory: "1024Mi"

to config.yaml

Once all the changes are made, run helm upgrade to switch your platform to the new config:

helm upgrade -f config.yaml $release_name . --namespace $namespace

Be sure to specify the platform namespace, not an Airflow namespace.

Infrastructure Estimates

To ensure plenty of room to run Airflow environments and hefty jobs, these estimates are going to be for a cluster with around 24CPUs in the US-East region.

AWS

ComponentItemYearly Cost (annual upfront pricing)
Compute3 t2.2xlarge (8 vCPU 32 GiB)$5640
EKS Control Plane$0.20 24 365$1752
Databasedb.t2.medium Postgres, Multi-AZ at $0.29/hr *24hr * 365$424
Total$7816

GCP

ComponentItemYearly Cost (annual upfront pricing)
Compute3 t2.2xlarge (8 vCPU 32 GiB) at $0.376/hr * 24hr * 365 days$6,291.17
Databasedb.t2.medium Postgres, Multi-AZ at $0.29/hr *24hr * 365$1397.98
Total$7689.15

For added customization, check out the GCP Pricing Calculator.

Azure

ComponentItemYearly Cost (annual upfront pricing)
Compute3 x D8s v3 (8 vCPU(s), 32 GiB)$8343.12
Database1 x Gen 5 (2 vCore), 25 GB Storage,LRS redundancy$1568
Total$9911.12

The Azure Price Calculator can be used to get an estimate for further customization.