Skip to main content
When you supply your own Airflow metadata database or result backend connection for a Deployment, Astronomer Software 0.37.x can write those credentials in plaintext to Houston API logs. Use this guide to check whether you’re affected, stop new log entries, and contain credentials that are already in your logs.

​
Check if you’re affected

You’re affected if you supply the database connection for one or more Deployments yourself, either in the metadataConnection, metadataConnectionJson, resultBackendConnection, or resultBackendConnectionJson arguments of the upsertDeployment mutation, or through a pre-deployment webhook. This guide covers the Houston API logs only. The credentials appear in the following locations:
  • The logs of the Houston API and Houston worker Pods in your platform namespace.
  • Any system that collects those Pod logs, such as a security information and event management (SIEM) system, a data lake, or a log archive in cold storage.
The platform Fluentd DaemonSet collects only Airflow component logs, so the Elasticsearch indices for Airflow logs don’t contain these credentials.

​
Step 1: Stop new log entries

Astro Private Cloud (APC) 2.1.1 redacts customer-managed database credentials written to log lines. Astronomer recommends that you upgrade to APC 2.1.1 or later. See Upgrade 0.37 to 2.x. Until you upgrade, set the Houston API log level to warn. The affected log lines use the info level, so this setting stops new entries. It also removes all other info log lines from the Houston API.
  1. Add the following to your values.yaml file:
  2. Apply the change. See Apply a config change.

​
Step 2: Restrict access to the logs

Limit who can read the affected logs while you rotate the credentials and remove the old entries.
  • In Kubernetes, restrict the get permission on pods/log in your platform namespace to the smallest group of users that need it. To check a user’s access, run kubectl auth can-i get pods/log -n <platform-namespace> --as <user>.
  • In every system that collects Houston API or Houston worker Pod logs, restrict read access to the indices, tables, or buckets that hold them.

​
Step 3: Rotate the exposed credentials

Because these credentials were stored in plaintext, Astronomer recommends that you rotate them. Changing the log level stops new entries, but credentials in existing entries remain valid until you change them. For each affected Deployment:
  1. In your database, set a new password for the database user, or create a new database user.
  2. Update the Deployment connection in the same location you set it originally: the upsertDeployment mutation or your pre-deployment webhook. See Use a MySQL or PostgreSQL database for metadata or storage.
  3. Confirm that the Deployment’s Airflow components restart and connect to the database.
  4. Remove the old password or the old database user.
At the info log level, the Houston API logs the new credentials when you update the Deployment. Complete Step 1 before you rotate the credentials.

​
Step 4: Purge the affected log entries

After you rotate the credentials, the old values in your logs no longer grant access. Remove or redact those entries anyway, so that a credential you missed during rotation doesn’t stay exposed.
  1. In every system that keeps Houston API or Houston worker Pod logs, search for entries that contain metadataConnection or resultBackendConnection. Start the search from the date you first supplied a connection for an affected Deployment.
  2. Delete the matching entries, or redact the credentials in them.
  3. Repeat the search in copies of these logs, such as SIEM archives, data lake tables, cold storage buckets, and backups.
Each system keeps these entries for as long as its retention policy allows. Include systems that you don’t search regularly, such as archives and backups.