> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Load platform secrets from mounted files (Preview)

> Deliver secrets to Astro Private Cloud platform components as mounted files instead of environment variables, and keep individual components on environment variables when you need to.

By default, the Astro Private Cloud (APC) platform Helm chart passes secrets, such as database connection strings and identity provider client secrets, to platform components as environment variables that reference Kubernetes Secrets with `valueFrom.secretKeyRef`. If your security policies don't allow secrets in environment variables, turn on secrets from files, and the chart mounts each secret as a file instead.

> [!NOTE]
>
> **Astro Private Cloud 2.2**
>
> This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

> [!NOTE]
>
> **Preview**
>
> Secrets from files is in Preview. Some platform components don't support it yet and keep their secrets in environment variables. See [Secrets that stay in environment variables](#secrets-that-stay-in-environment-variables).

## How it works

The feature covers two kinds of components:

* Components that run a *secrets loader*, code in the APC image that reads the secret files when the container starts and passes each value to the application. These are the APC API, APC Worker, Navigator, DP-Link, the data plane internal API, Pilot, the Prometheus service discovery reloader, and the database bootstrapper.
* Third-party components that read secret files with their own software: the Vector logging sidecar, the external Elasticsearch proxy, the Prometheus federation authentication proxy, PostgreSQL, and the PostgreSQL exporter.

For a component with a secrets loader, the chart:

* Mounts each of the component's secrets as a file, most of them in `/etc/astronomer/secrets/`, with mode `0440`.
* Sets the Pod's `fsGroup` to `global.secretsFromFiles.fsGroup`, which defaults to `1000`, so that the component's non-root process can read the files. On OpenShift, the chart doesn't set an `fsGroup`, because OpenShift assigns one to each namespace.
* Removes the environment variables that referenced those secrets.

Secrets loaders are strict. A component that the feature applies to gets its secrets only from files:

* `helm upgrade` fails if your values would also put one of the component's secrets in its environment, for example in the component's `env` list. The upgrade fails before it changes anything in your cluster.
* When the container starts, the loader exits with an error if one of the component's secrets is also in its environment, or if a file that it needs is missing or unreadable. Most loaders also exit on an empty file. See [Fix startup errors](#fix-startup-errors) for the exceptions.

The third-party components have their own checks:

* The Vector logging sidecar, the external Elasticsearch proxy, and the federation authentication proxy don't start if a secret file is missing or empty.
* PostgreSQL ignores an empty password file after its data folder exists, and its entrypoint passes the password to the database server in the server's environment.
* The PostgreSQL exporter's init container refuses an empty connection string. If you use the split datasource, the exporter doesn't check for an empty password file.

Secrets from files keeps the values out of each container's configured environment, so they don't appear in the output of `kubectl exec <pod-name> -- env`. After a loader reads a file, it still passes the value to its application in the process's own environment, because the application reads its configuration from there. Processes that the application starts inherit those values.

> [!NOTE]
>
> Components read their secret files when they start. After you change a Kubernetes Secret, restart the Pods that use it, as you would for a secret in an environment variable.

## Components and settings

Each component has its own `secretsFromFiles.enabled` setting. If you don't set it, the component follows `global.secretsFromFiles.enabled`. The database bootstrapper is the exception: its setting, `global.dbBootstrapper.secretsFromFiles.enabled`, defaults to `false` and doesn't follow the global setting.

| Component | Setting | Secrets it reads from files |
| - | - | - |
| APC API, APC Worker, and their Jobs and CronJobs | `astronomer.houston.secretsFromFiles.enabled` | Their database connections, the APC API's registry authentication header, and each entry in `astronomer.houston.secret` |
| Navigator | `astronomer.navigator.secretsFromFiles.enabled` | Its database connection and each entry in `astronomer.navigator.secret` |
| DP-Link | `astronomer.dpLink.secretsFromFiles.enabled` | Its database connections and each entry in `astronomer.dpLink.secret` |
| Data plane internal API | `astronomer.commander.secretsFromFiles.enabled` | Its database connections |
| Pilot | `astronomer.pilot.secretsFromFiles.enabled` | The Flightdeck database connection, when Flightdeck is on |
| Prometheus service discovery reloader, the `filesd-reloader` container in the control plane Prometheus Pod | `prometheus.filesdReloader.secretsFromFiles.enabled` | Its database connection |
| Database bootstrapper init containers in the APC API, APC Worker, data plane internal API, Grafana, and Laminar Pods, and in the APC API's Helm hook Jobs | `global.dbBootstrapper.secretsFromFiles.enabled` | The bootstrap database connection |
| Vector logging sidecar in the APC API and APC Worker Pods | `astronomer.houston.logging.loggingSidecar.secretsFromFiles.enabled` | The credentials for its log destination |
| External Elasticsearch proxy | `global.customLogging.secretsFromFiles.enabled` | The Elasticsearch credentials and AWS access keys |
| Prometheus federation authentication proxy on the data plane | `prometheus.federation.auth.secretsFromFiles.enabled` | Its authentication token |
| PostgreSQL | `postgresql.secretsFromFiles.enabled` | Its passwords, including the password for its metrics sidecar |
| PostgreSQL exporter | `prometheus-postgres-exporter.secretsFromFiles.enabled` | Its database connection. See the PostgreSQL exporter step in [Turn on secrets from files](#turn-on-secrets-from-files). |

### Secrets that stay in environment variables

The following secrets have no `secretsFromFiles` setting, and stay in environment variables whatever you set:

* Grafana's database connection
* The registry's notification header
* The secrets of [Laminar](https://astronomer.io/docs/astro-private-cloud/v-2-x/laminar-overview.md)
* Entries that you add to `astronomer.mcpServer.secret`. The [Astro Private Cloud MCP server](https://astronomer.io/docs/astro-private-cloud/v-2-x/apc-mcp-server.md) has no secrets of its own, and the chart doesn't refuse these entries while the feature is on.

The database bootstrapper init containers in the Grafana and Laminar Pods follow `global.dbBootstrapper.secretsFromFiles.enabled`.

Environment variables that you add to a component's own values also stay environment variables when the component has no `secretsFromFiles` setting. For example, registry storage credentials in `astronomer.registry.extraEnvVars` and Vector credentials in `vector.extraEnv` stay environment variables.

Secrets from files doesn't change Airflow Deployments. To keep Airflow connections and variables out of environment variables, use a [secrets backend](https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend).

## Prerequisites

* Astro Private Cloud 2.2 or later.
* Permission to run `helm upgrade` on your platform release. In a split installation, the control plane and each data plane are separate Helm releases. See [Apply a config change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md).
* The platform images that your APC chart version pins. Images from versions earlier than 2.2 can't read secrets from files. If you override an image for a component that the feature covers, use an image from the same APC release as the chart.

## Turn on secrets from files

**1. Move secrets out of environment variable lists**

While the feature applies to a component, `helm upgrade` fails if one of the following lists in your `values.yaml` file contains one of the component's secrets:

* `astronomer.houston.env`, `astronomer.navigator.env`, and `astronomer.dpLink.env`
* `astronomer.commander.env` and `astronomer.pilot.env`
* `prometheus.filesdReloader.extraEnv`
* `global.customLogging.extraEnv`
* `postgresql.extraEnv`

The chart also refuses an entry that sets a variable it sets for the component's secrets loader, such as `HOUSTON_SECRETS_FROM_FILES` or a `<NAME>_FILE` path. In `astronomer.dpLink.env`, `astronomer.pilot.env`, and the `extraEnv` lists, it refuses any entry that takes its value from a Kubernetes Secret.

To give the APC API, Navigator, or DP-Link a secret of your own, add it to the component's `secret` list instead of its `env` list. The chart delivers each entry as a file named after its `envName`, and the secrets loader requires that file. For example:

```yaml wrap theme={null}
astronomer:
  houston:
    secret:
      - envName: EMAIL__SMTP_URL
        secretName: astronomer-smtp
        secretKey: connection
```

If you store your external Elasticsearch credentials inline in `global.customLogging.secret`, store them under the `elastic` key of a Kubernetes Secret instead, and name the Secret in `global.customLogging.secretName`. See [Save your Elastic Cloud deployment credentials](https://astronomer.io/docs/astro-private-cloud/v-2-x/export-task-logs.md#save-your-elastic-cloud-deployment-credentials).

For entries in the other lists, remove the entry. The chart delivers those components' secrets itself.

**2. (Optional) Check the PostgreSQL exporter's connection string**

Skip this step unless `global.prometheusPostgresExporter.enabled` is `true` in your values.

The exporter can't read a whole connection string from a file. With the feature on, an init container named `split-data-source-name` reads the connection string from `prometheus-postgres-exporter.config.datasource.connectionSecret` and writes the database user, password, and address to separate files that the exporter reads. Only the init container mounts the Kubernetes Secret that holds the connection string.

The init container fails, and the exporter doesn't start, if the connection string:

* Is missing or empty.
* Isn't a `postgres://` or `postgresql://` URL. For example, a `host=... user=...` connection string.
* Contains a comma. Percent-encode it as `%2C`.
* Has an invalid percent-escape or a control character, or a user or password that starts or ends with whitespace.
* Has no user or no host.

The init container's error describes the problem and never includes the connection string. To read it, run:

```bash wrap theme={null}
kubectl logs -n <platform-namespace> <exporter-pod-name> -c split-data-source-name
```

If you already use the split datasource, with `connectionSecret` set to `null` and a `passwordSecret`, it keeps working, and the exporter reads its password from a file.

To keep the exporter on environment variables instead, turn the feature off for it:

```yaml wrap theme={null}
prometheus-postgres-exporter:
  secretsFromFiles:
    enabled: false
```

**3. Turn on the feature**

Add the following values to your `values.yaml` file:

```yaml wrap theme={null}
global:
  secretsFromFiles:
    enabled: true
  dbBootstrapper:
    secretsFromFiles:
      enabled: true
```

`global.secretsFromFiles.enabled` turns on the feature for every component that doesn't set its own setting to `false`. The database bootstrapper needs its own setting, as [Components and settings](#components-and-settings) explains.

In a split installation, add these values to the values file of each release, the control plane and each data plane, where you want the feature.

Change `global.secretsFromFiles.fsGroup` only if a component's image runs with a group other than `1000`.

**4. (Optional) Keep a component on environment variables**

To keep one component on environment variables while the feature is on for the rest of the platform, set the component's own setting to `false`. For example:

```yaml wrap theme={null}
astronomer:
  navigator:
    secretsFromFiles:
      enabled: false
```

To turn on the feature for some components only, leave `global.secretsFromFiles.enabled` set to `false`, and set those components' settings to `true`.

Each `secretsFromFiles.enabled` setting must be `true`, `false`, or unset. A quoted value, such as `"false"`, fails the render.

**5. Apply the change**

Apply your updated `values.yaml` file with `helm upgrade`. See [Apply a config change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md).

If the chart refuses your values, `helm upgrade` fails before it changes anything in your cluster, and the error names the setting to change. See [Fix Helm upgrade errors](#fix-helm-upgrade-errors).

The Pods of each component that the change affects restart.

**6. Verify the change**

1. List the environment variables that still take their values from Kubernetes Secrets:

   ```bash wrap theme={null}
   kubectl get pods -n <platform-namespace> -o json | jq -r '.items[] | .metadata.name as $pod | ((.spec.initContainers // []) + .spec.containers)[] | .name as $container | (.env // [])[] | select(.valueFrom.secretKeyRef) | "\($pod) \($container) \(.name)"'
   ```

   The command lists the Pod, container, and variable name for each one, and never a value. With the feature on for every covered component, the output lists only the [secrets that stay in environment variables](#secrets-that-stay-in-environment-variables), such as `GF_DATABASE_URL` in the Grafana Pod and `REGISTRY_NOTIFICATIONS_ENDPOINTS_0_HEADERS` in the registry Pod. In a split installation, run the command on each cluster.

2. Confirm that a component loaded its secrets. For example, for the APC API:

   ```bash wrap theme={null}
   kubectl logs -n <platform-namespace> deployment/<platform-release-name>-houston -c houston | grep 'secret(s) from files'
   ```

   The output looks similar to the following:

   ```text theme={null}
   [secrets-from-files] Done. Loaded 4 secret(s) from files.
   ```

   The number depends on your configuration. The secrets loader logs this line when the container starts, so the line can be missing from the log of a container that has run for a long time.

## Turn off secrets from files

Set `global.secretsFromFiles.enabled` to `false`. If you turned on `global.dbBootstrapper.secretsFromFiles.enabled`, set it to `false` too. Then [apply the change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md).

The components go back to environment variables that reference the same Kubernetes Secrets. A component whose own setting is `true` keeps reading its secrets from files.

## Troubleshoot

### Fix Helm upgrade errors

When the chart refuses your values, the error starts with a token for the problem, then names the setting to change:

| Token | Cause | Fix |
| - | - | - |
| `secret-in-environment` | A value puts one of a component's secrets in its environment. For example, an `env` entry sets one of the component's secrets, a list that the chart renders as written takes a value from a Kubernetes Secret, or `global.customLogging.secret` holds an inline credential. | Move the secret to the component's `secret` list, remove the entry, or turn the feature off for the component. |
| `loader-setting-in-environment` | An `env` entry sets a variable that the chart sets for the component's secrets loader. | Remove the entry. |
| `duplicate-secret-name` | Two `secret` entries have the same `envName`, or an `envName` names a secret that the chart already delivers to the component. | Give each entry a unique `envName`. |
| `invalid-secret-entry` | A `secret` entry has no `envName`. | Add an `envName` to the entry. |
| `invalid-gate-value` | A `secretsFromFiles.enabled` setting isn't `true`, `false`, or unset. For example, it's the quoted string `"false"`. | Set an unquoted `true` or `false`. |
| `secret-file-missing` | The PostgreSQL exporter's split datasource sets neither `passwordSecret` nor `password`. | Set `passwordSecret`. |

### Fix startup errors

When a secrets loader finds a problem, the container exits, and the Pod restarts with the `CrashLoopBackOff` status. The container's log lists each problem on its own line, with the variable name and a token, and ends with a line that describes how to fix the problems. For example:

```text theme={null}
[secrets-from-files] Refusing to start: 1 problem(s) with secrets while HOUSTON_SECRETS_FROM_FILES is on:
  - EMAIL__SMTP_URL (secret-in-environment): set in the environment, but it must come from a file while HOUSTON_SECRETS_FROM_FILES is on.
Fix: deliver each secret as a file, remove it from the container's environment (chart values astronomer.houston.env, astronomer.navigator.env, astronomer.dpLink.env), or turn file delivery off for this component (astronomer.houston|navigator|dpLink.secretsFromFiles.enabled=false).
```

To read the log of a container that restarted, run:

```bash wrap theme={null}
kubectl logs -n <platform-namespace> <pod-name> -c <container-name> --previous
```

The exact wording differs between components, but each loader uses the same tokens:

| Token | Cause |
| - | - |
| `secret-in-environment` | One of the component's secrets is also in its environment. For example, a mutating admission webhook or a manual edit of the Pod's controller added it. |
| `secret-on-command-line` | A secret is passed as a command-line argument. The data plane internal API and the database bootstrapper check for this. |
| `secret-file-missing` | The file for a secret that the component requires doesn't exist. |
| `secret-file-unreadable` | The component can't read a file. For example, the Pod's `fsGroup` doesn't give the process access to it, or it isn't valid UTF-8. |
| `secret-file-empty` | A secret's file is empty. The APC API, APC Worker, Navigator, and DP-Link treat an empty file as an error only for their database connections, and log a warning for other secrets. |
| `invalid-gate-value` | The variable that turns on the component's secrets loader has a value other than `true` or `false`. |

If the cause is a permission error, check that the Pod's `securityContext.fsGroup` is set, and that no admission policy in your cluster removes or replaces it.