Astro Private Cloud (APC) uses PostgreSQL as the primary database system for both platform management (the APC API) and Apache Airflow deployments. The following sections describe the database architecture, provisioning, connection pooling, and high availability configuration.
The APC API uses a PostgreSQL database to store platform configuration and state.
Key tables:
Each Airflow Deployment gets its own isolated PostgreSQL database containing two schemas:
The database name is derived from the Deployment release name:
Each schema has a separate database user with permissions scoped to that schema. the deployment orchestrator provisions the database and both schemas automatically when you create a Deployment. For more information about what Airflow stores in the metadata database, see Understanding the Airflow metadata database.
By default, the deployment orchestrator automatically provisions databases for new Deployments. No additional configuration is required.
To use pre-existing or managed databases, set skipAirflowDatabaseProvisioning to true in the upsertDeployment mutation:
When using external databases, provide the connection string in your Deployment configuration. For complete setup steps with connection string examples, see Bring your own Airflow database. For supported PostgreSQL versions, see the Apache Airflow database backend documentation.
APC uses PgBouncer for connection pooling to reduce database connection overhead. Airflow can open many database connections due to its distributed nature, and each PostgreSQL connection creates a dedicated OS process. PgBouncer reduces this overhead by maintaining a pool of reusable server connections. For more detail on why PgBouncer is recommended, see the Apache Airflow Helm chart production guide.
PgBouncer is only used with the embedded PostgreSQL database. When you provide a metadataConnectionString URI through the upsertDeployment mutation, Airflow connects directly to the external database and bypasses PgBouncer. If you use an external database and need connection pooling, configure it through your managed database service or deploy a separate PgBouncer instance.
PgBouncer operates in transaction pool mode by default, which means server connections are returned to the pool after each transaction completes. This mode doesn’t support session-level features such as prepared statements or SET commands that persist across transactions.
Pool sizes are configured per Deployment in the Airflow chart values:
These defaults match the Apache Airflow Helm chart defaults and work for most Deployments. To determine whether you need to adjust pool sizes, monitor PgBouncer’s cl_waiting metric. If clients consistently wait for connections, increase pool sizes or scale PgBouncer replicas.
Airflow also maintains its own SQLAlchemy connection pool (default pool_size: 5, max_overflow: 10) between each Airflow component and PgBouncer. With these defaults, each Airflow process can open up to 15 simultaneous database connections. The full connection chain is: Airflow component → SQLAlchemy pool → PgBouncer → PostgreSQL.
PgBouncer in APC includes Kerberos support:
For complete Kerberos database setup instructions, see Configure Kerberos authentication for Airflow databases.
Astronomer doesn’t recommend using internal PostgreSQL instance. If you need production database resiliency, use an externally managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL. Managed services provide built-in replication, automated failover, and backup capabilities that are more reliable than running replication within the Helm chart.
Replication is disabled by default. The following example shows a recommended production configuration that enables streaming replication with synchronous commit:
The following table compares the chart default values with starting points that Astronomer recommends for production environments. The Default column reflects the values shipped in the Helm chart. The Recommended for production column reflects general guidance based on Astronomer operational experience. Adjust these values based on the number of Deployments, workload scale, and observed resource utilization in your environment.
For additional production guidance, see the Apache Airflow Helm chart production guide. Astronomer also recommends using a managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL rather than the embedded PostgreSQL container for production workloads.
These values match the Apache Airflow Helm chart defaults. No official formula ties pool sizes to a specific workload metric such as Dag count. Instead, adjust pool sizes based on observed connection utilization.
For database backup and restore procedures, including size estimation, pg_dump/mysqldump commands, and restore steps, see Access Airflow database.
The APC database connection depends on whether you use an external database or the in-cluster PostgreSQL. You configure this during control plane installation.
For external databases, provide the connection through houston.backendConnection in your Helm values:
Alternatively, provide a pre-existing Kubernetes secret name through houston.backendSecretName. The secret must contain a connection key with the full connection URI.
To retrieve the active APC database connection string, read the astronomer-bootstrap secret:
The in-cluster PostgreSQL option (global.postgresql.enabled: true) is only for development or proof-of-concept environments and isn’t supported in production.
Both the metadata and result backend connections use the same database with different schemas and credentials:
When PgBouncer is enabled, Airflow components connect through PgBouncer on port 6543 instead of directly to PostgreSQL on port 5432.
To retrieve the connection string for a specific Deployment, see Access Airflow database.
Monitor the following metrics to track database and connection pool health:
cl_waiting and database size growth.