> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Respond to Airflow database credentials in APC logs

> Find out if your Astro Private Cloud version wrote Airflow metadata database credentials to logs, stop new entries, and contain the credentials that are already exposed.

When you supply your own Airflow metadata database or result backend connection for a Deployment, some Astro Private Cloud (APC) versions can write those credentials in plaintext to APC API logs and audit events. Use this guide to check whether you're affected, stop new log entries, and contain credentials that are already in your logs.

## Check if you're affected

You're affected if both of the following are true:

* Your installation runs, or ran, an affected APC version.
* You supply the database connection for one or more Deployments yourself, either in the `metadataConnection`, `metadataConnectionJson`, `resultBackendConnection`, or `resultBackendConnectionJson` arguments of the `upsertDeployment` mutation, or through a pre-deployment webhook.

This guide covers the APC API logs and audit events only.

| APC version | Status | Action |
| - | - | - |
| 2.1.1 and later | Credentials redacted from log lines | Contain the credentials that earlier versions wrote to your logs. |
| 2.0.x and 2.1.0 | Affected | Upgrade to 2.1.1 or later, then contain the exposed credentials. |
| 1.0.x, 1.1.x, and 1.2.x | Affected | Set the APC API log level to `warn`, contain the exposed credentials, and plan an upgrade to 2.1.1 or later. |
| 0.37.x | Affected | Set the APC API log level to `warn`, contain the exposed credentials, and plan an upgrade to 2.1.1. |

The credentials appear in the following locations:

* The logs of the APC API and APC Worker Pods in your platform namespace.
* Any system that collects those Pod logs, such as a security information and event management (SIEM) system, a data lake, or a log archive in cold storage.
* APC [audit events](/docs/astro-private-cloud/v-2-x/audit-logging-overview), including events for failed requests, and the sink that receives them.

The APC Vector DaemonSet doesn't collect APC API logs, so the Elasticsearch indices for Airflow logs don't contain these credentials.

## Step 1: Stop new log entries

Upgrade to APC 2.1.1 or later. APC 2.1.1 redacts customer-managed database credentials written to log lines. No 2.0.x release includes the fix. For details, see the [APC 2.1.1 release notes](/docs/astro-private-cloud/v-2-x/release-notes).

* To upgrade from 2.0.x or 2.1.0, see [Upgrade 2.x to 2.x](/docs/astro-private-cloud/v-2-x/upgrade-apc-minor-version).
* To upgrade from 1.x, see [Upgrade 1.x to 2.x](/docs/astro-private-cloud/v-2-x/upgrade-1x-to-2).
* To upgrade from 0.37, see [Upgrade 0.37 to 2.x](/docs/astro-private-cloud/v-2-x/upgrade-037-to-2).

If you run 2.0.x or 2.1.0 and can't upgrade yet, contact [Astronomer support](https://support.astronomer.io).

If you run 1.0.x, 1.1.x, 1.2.x, or 0.37.x and can't upgrade yet, set the APC API log level to `warn`. The affected log lines use the `info` level, so this setting stops new entries. It also removes all other `info` log lines from the APC API. Add the following to your `values.yaml` file, then [apply the config change](/docs/astro-private-cloud/v-2-x/apply-platform-config):

```yaml theme={null}
astronomer:
  houston:
    config:
      logging:
        level: warn
```

## Step 2: Restrict access to the logs

Limit who can read the affected logs while you rotate the credentials and remove the old entries.

* In Kubernetes, restrict the `get` permission on `pods/log` in your platform namespace to the smallest group of users that need it. To check a user's access, run `kubectl auth can-i get pods/log -n <platform-namespace> --as <user>`.
* In every system that collects APC API or APC Worker Pod logs, restrict read access to the indices, tables, or buckets that hold them.
* Restrict read access to the sink that receives your audit events.

## Step 3: Rotate the exposed credentials

Because these credentials were stored in plaintext, Astronomer recommends that you rotate them. An upgrade or log level change affects only new log entries. Credentials in existing entries remain valid until you change them.

For each affected Deployment:

1. In your database, set a new password for the database user, or create a new database user.
2. Update the Deployment connection in the same location you set it originally: the `upsertDeployment` mutation or your pre-deployment webhook. See [Use a MySQL or PostgreSQL database for metadata or storage](/docs/astro-private-cloud/v-2-x/multi-db#step-6-add-to-deployment-configuration).
3. Confirm that the Deployment's Airflow components restart and connect to the database.
4. Remove the old password or the old database user.

If a Deployment uses [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover), rotate the credentials for its database users on both data plane clusters.

<Warning>
  On an affected version, the APC API logs the new credentials when you update the Deployment. Complete [Step 1](#step-1-stop-new-log-entries) first if you can do it quickly. If you can't, rotate the credentials now, and contain the new log entries in the same way as the old ones.
</Warning>

## Step 4: Purge the affected log entries

After you rotate the credentials, the old values in your logs no longer grant access. Remove or redact those entries anyway, so that a credential you missed during rotation doesn't stay exposed.

1. In every system that keeps APC API or APC Worker Pod logs, search for entries that contain `metadataConnection` or `resultBackendConnection`. Start the search from the date you first supplied a connection for an affected Deployment.
2. Delete the matching entries, or redact the credentials in them.
3. Do the same in the sink that receives your audit events.
4. Repeat the search in copies of these logs, such as SIEM archives, data lake tables, cold storage buckets, and backups.

Each system keeps these entries for as long as its retention policy allows. Include systems that you don't search regularly, such as archives and backups.
