Load platform secrets from mounted files (Preview)

By default, the Astro Private Cloud (APC) platform Helm chart passes secrets, such as database connection strings and identity provider client secrets, to platform components as environment variables that reference Kubernetes Secrets with valueFrom.secretKeyRef. If your security policies don’t allow secrets in environment variables, turn on secrets from files, and the chart mounts each secret as a file instead.

Astro Private Cloud 2.2This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

PreviewSecrets from files is in Preview. Some platform components don’t support it yet and keep their secrets in environment variables. See Secrets that stay in environment variables.

How it works

The feature covers two kinds of components:

Components that run a secrets loader, code in the APC image that reads the secret files when the container starts and passes each value to the application. These are the APC API, APC Worker, Navigator, DP-Link, the data plane internal API, Pilot, the Prometheus service discovery reloader, and the database bootstrapper.
Third-party components that read secret files with their own software: the Vector logging sidecar, the external Elasticsearch proxy, the Prometheus federation authentication proxy, PostgreSQL, and the PostgreSQL exporter.

For a component with a secrets loader, the chart:

Mounts each of the component’s secrets as a file, most of them in /etc/astronomer/secrets/, with mode 0440.
Sets the Pod’s fsGroup to global.secretsFromFiles.fsGroup, which defaults to 1000, so that the component’s non-root process can read the files. On OpenShift, the chart doesn’t set an fsGroup, because OpenShift assigns one to each namespace.
Removes the environment variables that referenced those secrets.

Secrets loaders are strict. A component that the feature applies to gets its secrets only from files:

helm upgrade fails if your values would also put one of the component’s secrets in its environment, for example in the component’s env list. The upgrade fails before it changes anything in your cluster.
When the container starts, the loader exits with an error if one of the component’s secrets is also in its environment, or if a file that it needs is missing or unreadable. Most loaders also exit on an empty file. See Fix startup errors for the exceptions.

The third-party components have their own checks:

The Vector logging sidecar, the external Elasticsearch proxy, and the federation authentication proxy don’t start if a secret file is missing or empty.
PostgreSQL ignores an empty password file after its data folder exists, and its entrypoint passes the password to the database server in the server’s environment.
The PostgreSQL exporter’s init container refuses an empty connection string. If you use the split datasource, the exporter doesn’t check for an empty password file.

Secrets from files keeps the values out of each container’s configured environment, so they don’t appear in the output of kubectl exec <pod-name> -- env. After a loader reads a file, it still passes the value to its application in the process’s own environment, because the application reads its configuration from there. Processes that the application starts inherit those values.

Components read their secret files when they start. After you change a Kubernetes Secret, restart the Pods that use it, as you would for a secret in an environment variable.

Components and settings

Each component has its own secretsFromFiles.enabled setting. If you don’t set it, the component follows global.secretsFromFiles.enabled. The database bootstrapper is the exception: its setting, global.dbBootstrapper.secretsFromFiles.enabled, defaults to false and doesn’t follow the global setting.

Component	Setting	Secrets it reads from files
APC API, APC Worker, and their Jobs and CronJobs	`astronomer.houston.secretsFromFiles.enabled`	Their database connections, the APC API’s registry authentication header, and each entry in `astronomer.houston.secret`
Navigator	`astronomer.navigator.secretsFromFiles.enabled`	Its database connection and each entry in `astronomer.navigator.secret`
DP-Link	`astronomer.dpLink.secretsFromFiles.enabled`	Its database connections and each entry in `astronomer.dpLink.secret`
Data plane internal API	`astronomer.commander.secretsFromFiles.enabled`	Its database connections
Pilot	`astronomer.pilot.secretsFromFiles.enabled`	The Flightdeck database connection, when Flightdeck is on
Prometheus service discovery reloader, the `filesd-reloader` container in the control plane Prometheus Pod	`prometheus.filesdReloader.secretsFromFiles.enabled`	Its database connection
Database bootstrapper init containers in the APC API, APC Worker, data plane internal API, Grafana, and Laminar Pods, and in the APC API’s Helm hook Jobs	`global.dbBootstrapper.secretsFromFiles.enabled`	The bootstrap database connection
Vector logging sidecar in the APC API and APC Worker Pods	`astronomer.houston.logging.loggingSidecar.secretsFromFiles.enabled`	The credentials for its log destination
External Elasticsearch proxy	`global.customLogging.secretsFromFiles.enabled`	The Elasticsearch credentials and AWS access keys
Prometheus federation authentication proxy on the data plane	`prometheus.federation.auth.secretsFromFiles.enabled`	Its authentication token
PostgreSQL	`postgresql.secretsFromFiles.enabled`	Its passwords, including the password for its metrics sidecar
PostgreSQL exporter	`prometheus-postgres-exporter.secretsFromFiles.enabled`	Its database connection. See the PostgreSQL exporter step in Turn on secrets from files.

Secrets that stay in environment variables

The following secrets have no secretsFromFiles setting, and stay in environment variables whatever you set:

Grafana’s database connection
The registry’s notification header
The secrets of Laminar
Entries that you add to astronomer.mcpServer.secret. The Astro Private Cloud MCP server has no secrets of its own, and the chart doesn’t refuse these entries while the feature is on.

The database bootstrapper init containers in the Grafana and Laminar Pods follow global.dbBootstrapper.secretsFromFiles.enabled. Environment variables that you add to a component’s own values also stay environment variables when the component has no secretsFromFiles setting. For example, registry storage credentials in astronomer.registry.extraEnvVars and Vector credentials in vector.extraEnv stay environment variables. Secrets from files doesn’t change Airflow Deployments. To keep Airflow connections and variables out of environment variables, use a secrets backend.

Prerequisites

Astro Private Cloud 2.2 or later.
Permission to run helm upgrade on your platform release. In a split installation, the control plane and each data plane are separate Helm releases. See Apply a config change.
The platform images that your APC chart version pins. Images from versions earlier than 2.2 can’t read secrets from files. If you override an image for a component that the feature covers, use an image from the same APC release as the chart.

Turn on secrets from files

Move secrets out of environment variable lists

While the feature applies to a component, helm upgrade fails if one of the following lists in your values.yaml file contains one of the component’s secrets:

astronomer.houston.env, astronomer.navigator.env, and astronomer.dpLink.env
astronomer.commander.env and astronomer.pilot.env
prometheus.filesdReloader.extraEnv
global.customLogging.extraEnv
postgresql.extraEnv

The chart also refuses an entry that sets a variable it sets for the component’s secrets loader, such as HOUSTON_SECRETS_FROM_FILES or a <NAME>_FILE path. In astronomer.dpLink.env, astronomer.pilot.env, and the extraEnv lists, it refuses any entry that takes its value from a Kubernetes Secret.To give the APC API, Navigator, or DP-Link a secret of your own, add it to the component’s secret list instead of its env list. The chart delivers each entry as a file named after its envName, and the secrets loader requires that file. For example:

astronomer:
  houston:
    secret:
      - envName: EMAIL__SMTP_URL
        secretName: astronomer-smtp
        secretKey: connection

If you store your external Elasticsearch credentials inline in global.customLogging.secret, store them under the elastic key of a Kubernetes Secret instead, and name the Secret in global.customLogging.secretName. See Save your Elastic Cloud deployment credentials.For entries in the other lists, remove the entry. The chart delivers those components’ secrets itself.

(Optional) Check the PostgreSQL exporter's connection string

Skip this step unless global.prometheusPostgresExporter.enabled is true in your values.The exporter can’t read a whole connection string from a file. With the feature on, an init container named split-data-source-name reads the connection string from prometheus-postgres-exporter.config.datasource.connectionSecret and writes the database user, password, and address to separate files that the exporter reads. Only the init container mounts the Kubernetes Secret that holds the connection string.The init container fails, and the exporter doesn’t start, if the connection string:

Is missing or empty.
Isn’t a postgres:// or postgresql:// URL. For example, a host=... user=... connection string.
Contains a comma. Percent-encode it as %2C.
Has an invalid percent-escape or a control character, or a user or password that starts or ends with whitespace.
Has no user or no host.

The init container’s error describes the problem and never includes the connection string. To read it, run:

kubectl logs -n <platform-namespace> <exporter-pod-name> -c split-data-source-name

If you already use the split datasource, with connectionSecret set to null and a passwordSecret, it keeps working, and the exporter reads its password from a file.To keep the exporter on environment variables instead, turn the feature off for it:

prometheus-postgres-exporter:
  secretsFromFiles:
    enabled: false

Turn on the feature

Add the following values to your values.yaml file:

global:
  secretsFromFiles:
    enabled: true
  dbBootstrapper:
    secretsFromFiles:
      enabled: true

global.secretsFromFiles.enabled turns on the feature for every component that doesn’t set its own setting to false. The database bootstrapper needs its own setting, as Components and settings explains.In a split installation, add these values to the values file of each release, the control plane and each data plane, where you want the feature.Change global.secretsFromFiles.fsGroup only if a component’s image runs with a group other than 1000.

(Optional) Keep a component on environment variables

To keep one component on environment variables while the feature is on for the rest of the platform, set the component’s own setting to false. For example:

astronomer:
  navigator:
    secretsFromFiles:
      enabled: false

To turn on the feature for some components only, leave global.secretsFromFiles.enabled set to false, and set those components’ settings to true.Each secretsFromFiles.enabled setting must be true, false, or unset. A quoted value, such as "false", fails the render.

Apply the change

Apply your updated values.yaml file with helm upgrade. See Apply a config change.If the chart refuses your values, helm upgrade fails before it changes anything in your cluster, and the error names the setting to change. See Fix Helm upgrade errors.The Pods of each component that the change affects restart.

Verify the change

List the environment variables that still take their values from Kubernetes Secrets:
kubectl get pods -n <platform-namespace> -o json | jq -r '.items[] | .metadata.name as $pod | ((.spec.initContainers // []) + .spec.containers)[] | .name as $container | (.env // [])[] | select(.valueFrom.secretKeyRef) | "\($pod) \($container) \(.name)"'
The command lists the Pod, container, and variable name for each one, and never a value. With the feature on for every covered component, the output lists only the secrets that stay in environment variables, such as GF_DATABASE_URL in the Grafana Pod and REGISTRY_NOTIFICATIONS_ENDPOINTS_0_HEADERS in the registry Pod. In a split installation, run the command on each cluster.
Confirm that a component loaded its secrets. For example, for the APC API:
kubectl logs -n <platform-namespace> deployment/<platform-release-name>-houston -c houston | grep 'secret(s) from files'
The output looks similar to the following:
[secrets-from-files] Done. Loaded 4 secret(s) from files.
The number depends on your configuration. The secrets loader logs this line when the container starts, so the line can be missing from the log of a container that has run for a long time.

Turn off secrets from files

Set global.secretsFromFiles.enabled to false. If you turned on global.dbBootstrapper.secretsFromFiles.enabled, set it to false too. Then apply the change. The components go back to environment variables that reference the same Kubernetes Secrets. A component whose own setting is true keeps reading its secrets from files.

Troubleshoot

Fix Helm upgrade errors

When the chart refuses your values, the error starts with a token for the problem, then names the setting to change:

Token	Cause	Fix
`secret-in-environment`	A value puts one of a component’s secrets in its environment. For example, an `env` entry sets one of the component’s secrets, a list that the chart renders as written takes a value from a Kubernetes Secret, or `global.customLogging.secret` holds an inline credential.	Move the secret to the component’s `secret` list, remove the entry, or turn the feature off for the component.
`loader-setting-in-environment`	An `env` entry sets a variable that the chart sets for the component’s secrets loader.	Remove the entry.
`duplicate-secret-name`	Two `secret` entries have the same `envName`, or an `envName` names a secret that the chart already delivers to the component.	Give each entry a unique `envName`.
`invalid-secret-entry`	A `secret` entry has no `envName`.	Add an `envName` to the entry.
`invalid-gate-value`	A `secretsFromFiles.enabled` setting isn’t `true`, `false`, or unset. For example, it’s the quoted string `"false"`.	Set an unquoted `true` or `false`.
`secret-file-missing`	The PostgreSQL exporter’s split datasource sets neither `passwordSecret` nor `password`.	Set `passwordSecret`.

Fix startup errors

When a secrets loader finds a problem, the container exits, and the Pod restarts with the CrashLoopBackOff status. The container’s log lists each problem on its own line, with the variable name and a token, and ends with a line that describes how to fix the problems. For example:

[secrets-from-files] Refusing to start: 1 problem(s) with secrets while HOUSTON_SECRETS_FROM_FILES is on:
  - EMAIL__SMTP_URL (secret-in-environment): set in the environment, but it must come from a file while HOUSTON_SECRETS_FROM_FILES is on.
Fix: deliver each secret as a file, remove it from the container's environment (chart values astronomer.houston.env, astronomer.navigator.env, astronomer.dpLink.env), or turn file delivery off for this component (astronomer.houston|navigator|dpLink.secretsFromFiles.enabled=false).

To read the log of a container that restarted, run:

kubectl logs -n <platform-namespace> <pod-name> -c <container-name> --previous

The exact wording differs between components, but each loader uses the same tokens:

Token	Cause
`secret-in-environment`	One of the component’s secrets is also in its environment. For example, a mutating admission webhook or a manual edit of the Pod’s controller added it.
`secret-on-command-line`	A secret is passed as a command-line argument. The data plane internal API and the database bootstrapper check for this.
`secret-file-missing`	The file for a secret that the component requires doesn’t exist.
`secret-file-unreadable`	The component can’t read a file. For example, the Pod’s `fsGroup` doesn’t give the process access to it, or it isn’t valid UTF-8.
`secret-file-empty`	A secret’s file is empty. The APC API, APC Worker, Navigator, and DP-Link treat an empty file as an error only for their database connections, and log a warning for other secrets.
`invalid-gate-value`	The variable that turns on the component’s secrets loader has a value other than `true` or `false`.

If the cause is a permission error, check that the Pod’s securityContext.fsGroup is set, and that no admission policy in your cluster removes or replaces it.

​How it works

​Components and settings

​Secrets that stay in environment variables

​Prerequisites

​Turn on secrets from files

​Turn off secrets from files

​Troubleshoot

​Fix Helm upgrade errors

​Fix startup errors

How it works

Components and settings

Secrets that stay in environment variables

Prerequisites

Turn on secrets from files

Turn off secrets from files

Troubleshoot

Fix Helm upgrade errors

Fix startup errors