Configure liveness and readiness probes
In Astronomer Software, you can create liveness and readiness probes to assess whether your Kubernetes Pods or network are healthy and can process requests.
Some components in Astronomer Software include liveness and readiness probes by default, but all components support adding and configuring them. Astronomer Software allows you to use the Kubernetes liveness and readiness probe definitions so you can monitor the state of your Pods.
Liveness probes can be useful in all cases. However, readiness probes might be most useful for the following scenarios:
- If you have network ports open on the container
- Containers that do not have open ports, but have multiple processes within the container
- A setup where a process in a container might never reach a
healthy
state because it is waiting for some state to be achieved.
Default probe behavior
You can use the following structure to define your probes in your values.yaml
file. For example, you might want to adjust any default values by configuring the amount of time until a timeout. You can refer to some of the existing Default probe configurations.
You can add any definitions that are compatible with Kubernetes probes. However, because Kubernetes does not allow having more than one handler of probes you must be sure that you do not define probes to use both exec
and httpGet
. For consistency, the examples shown in the Default Astronomer Helm probe configurations use httpGet
, but can use exec
when appropriate.
Liveness probe templates
livenessProbe:
enabled: true
httpGet:
path: /index.html
port: 443
Readiness probe templates
readinessProbe:
enabled: true
httpGet:
path: /index.html
port: 444
Retrieve existing probe definitions
You can retrieve the default probe definitions from the Kubernetes manifest. The following example shows how to retrieve the definitions for Houston.
kubectl -n "${NAMESPACE}" get deployment -l component=houston -o yaml
This command produces a large amount of yaml output describing your Houston configuration. Within this output, is a section describing the livenessProbe
, which looks like the following:
livenessProbe:
failureThreshold: 10
httpGet:
path: /v1/healthz
port: 8871
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
You can copy and paste this output into your values.yaml
file for your Houston configuration, then adjust the values you want to customize. Then apply a platform config change.
Reference Helm values within your probes
Because values for the liveness and readiness probes are passed through the Helm template function, you can reference Helm values within the probes. Specifically, the livenessProbe
and readiness
values are rendered to yaml, then passed through the Helm template function, which renders any Helm template syntaxes into the produced yaml.
For example, instead of hardcoding values for your probes to match values defined by other configurations in your values.yaml
file, you can use the configuration variable itself.
The following example, using the alertsmanager
yaml configuration, shows how the path and ports are defined by Values.ports.http
and Values.prefixURL
elsewhere in the values.yaml
file.
readinessProbe:
httpGet:
path: {{ .Values.prefixURL }}/#/status
port: {{ .Values.ports.http }}
initialDelaySeconds: 30
timeoutSeconds: 30
Default Astronomer Helm probe configurations
The following components have their default probe configuration defined in the Astronomer Helm chart.
If a component does not have probes defined by default, you can see which options can have custom probe configurations.
Alert manager
alertmanager_auth-proxy:
livenessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
The following can also be configured to include liveness and readiness probes:
alertmanager:
livenessProbe: {}
Astronomer
astronomer:
astroUI:
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
commander:
livenessProbe:
httpGet:
path: /healthz
port: 8880
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 5
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /healthz
port: 8880
initialDelaySeconds: 10
periodSeconds: 10
houston:
livenessProbe:
httpGet:
path: /v1/healthz
port: 8871
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 10
readinessProbe:
httpGet:
path: /v1/healthz
port: 8871
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 10
registry:
livenessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
The following can also be configured to include liveness and readiness probes:
astronomer:
configSyncer:
livenessProbe: {}
readinessProbe: {}
houston:
bootstrapper:
livenessProbe: {}
readinessProbe: {}
cleanupAirflowDb:
livenessProbe: {}
readinessProbe: {}
cleanupDeployRevisions:
livenessProbe: {}
readinessProbe: {}
cleanupDeployments:
livenessProbe: {}
readinessProbe: {}
dbMigration:
livenessProbe: {}
readinessProbe: {}
taskUsageMetrics:
livenessProbe: {}
readinessProbe: {}
updateCheck:
livenessProbe: {}
readinessProbe: {}
updateResourceStrategy:
livenessProbe: {}
readinessProbe: {}
updateRuntimeCheck:
livenessProbe: {}
readinessProbe: {}
upgradeDeployments:
livenessProbe: {}
readinessProbe: {}
waitForDB:
livenessProbe: {}
readinessProbe: {}
worker:
livenessProbe: {}
readinessProbe: {}
Elasticsearch
elasticsearch:
client:
livenessProbe:
httpGet:
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 90
readinessProbe:
httpGet:
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 5
exporter:
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 30
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 10
timeoutSeconds: 10
master:
livenessProbe:
tcpSocket:
port: 9300
readinessProbe:
httpGet:
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 5
The following components do not have probes configured by default:
elasticsearch:
curator:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
data:
readinessProbe:
exec:
command:
- /bin/true
nginx:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
sysctlInitContainer:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
External-es-proxy
The following components do not have probes configured by default:
external-es-proxy:
awsproxy:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
Fluentd
livenessProbe:
exec:
command:
- /bin/bash
- -c
- >-
if (( $(ruby -e "require 'net/http';require 'uri';uri = URI.parse('http://127.0.0.1:24231/metrics');response = Net::HTTP.get_response(uri);puts response.body" | grep 'fluentd_output_status_buffer_queue_length{' | awk '{ print ($NF > 8) }') )); then exit 1; fi; exit 0
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 5
The following components do not have probes configured by default:
fluentd:
readinessProbe:
exec:
command:
- /bin/true
Global
The following components do not have probes configured by default:
global:
loggingSidecar:
readinessProbe: {}
livenessProbe: {}
dagOnlyDeployment:
server:
readinessProbe: {}
livenessProbe: {}
client:
readinessProbe: {}
livenessProbe: {}
authSidecar:
readinessProbe: {}
livenessProbe: {}
Grafana
grafana:
livenessProbe:
httpGet:
path: /api/health
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /api/health
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
The following components do not have probes configured by default:
grafana:
bootstrapper:
livenessProbe: {}
readinessProbe: {}
waitForDB:
livenessProbe: {}
readinessProbe: {}
Kibana
kibana:
livenessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
The following components do not have probes configured by default:
kibana:
defaultIndexJob:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
Kube-state
The following components do not have probes configured by default:
kube-state:
readinessProbe: {}
NATS
nats:
nats:
livenessProbe:
httpGet:
path: /
port: 8222
initialDelaySeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8222
initialDelaySeconds: 10
timeoutSeconds: 5
The following components do not have probes configured by default:
nats:
exporter:
enabled: true
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
reloader:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
nginx
The following components do not have probes configured by default:
defaultBackend:
readinessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
PgBouncer
pgbouncer:
livenessProbe:
tcpSocket:
port: 5432
readinessProbe:
tcpSocket:
port: 5432
PostgreSQL
livenessProbe:
exec:
command:
- sh
- -c
- exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
exec:
command:
- sh
- -c
- -e
- 'pg_isready -U "postgres" -h 127.0.0.1 -p 5432\n'
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
The following components do not have probes configured by default:
postgresql:
metrics:
livenessProbe: {}
readinessProbe: {}
Prometheus
prometheus:
livenessProbe:
httpGet:
path: /-/healthy
port: 9090
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /-/ready
port: 9090
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 1
authproxy:
livenessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /healthz
port: 8084
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
prometheus-blackbox-exporter:
livenessProbe:
httpGet:
path: /health
port: http
readinessProbe:
httpGet:
path: /health
port: http
prometheus-node-exporter:
livenessProbe:
httpGet:
path: /
port: 9100
readinessProbe:
httpGet:
path: /
port: 9100
prometheus-postgres-exporter:
livenessProbe:
tcpSocket:
port: 9187
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
tcpSocket:
port: 9187
initialDelaySeconds: 5
periodSeconds: 10
The following components do not have probes configured by default:
prometheus:
configMapReloader:
livenessProbe: {}
readinessProbe: {}
filesdReloader:
livenessProbe: {}
readinessProbe: {}
STAN
livenessProbe:
httpGet:
path: /streaming/serverz
port: monitor
initialDelaySeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /streaming/serverz
port: monitor
initialDelaySeconds: 10
timeoutSeconds: 5
The following components do not have probes configured by default:
stan:
exporter:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
waitForNatsServer:
livenessProbe:
exec:
command:
- /bin/true
readinessProbe:
exec:
command:
- /bin/true
Default Airflow chart probe configurations
You can also define liveness and readiness probes using the Astronomer Airflow chart.
Airflow
This includes:
dagProcessor
flower
pgbouncer
postgresql
scheduler
triggerer
webserver
workers
airflow:
dagProcessor:
livenessProbe:
command: null
failureThreshold: 5
initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 20
flower:
livenessProbe:
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
readinessProbe:
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
pgbouncer:
metricsExporterSidecar:
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
readinessProbe:
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
postgresql:
metrics:
customLivenessProbe: {}
customReadinessProbe: {}
livenessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
primary:
customLivenessProbe: {}
customReadinessProbe: {}
livenessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readReplicas:
customLivenessProbe: {}
customReadinessProbe: {}
livenessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
scheduler:
livenessProbe:
command: null
failureThreshold: 5
initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
triggerer:
livenessProbe:
command: null
failureThreshold: 5
initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 20
webserver:
livenessProbe:
failureThreshold: 5
initialDelaySeconds: 15
periodSeconds: 10
scheme: HTTP
timeoutSeconds: 5
readinessProbe:
failureThreshold: 5
initialDelaySeconds: 15
periodSeconds: 10
scheme: HTTP
timeoutSeconds: 5
workers:
livenessProbe:
command: null
enabled: true
failureThreshold: 5
initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 20
Auth sidecar
The following can also be configured to include liveness and readiness probes:
authSidecar:
livenessProbe: {}
readinessProbe: {}
DAG deploy server
The following can also be configured to include liveness and readiness probes:
dagDeploy:
livenessProbe: {}
readinessProbe: {}
Logging sidecar
The following can also be configured to include liveness and readiness probes:
loggingSidecar:
livenessProbe: {}
readinessProbe: {}