You can create Astronomer Software Deployments with the Houston API that use pre-created databases, external to the Airflow Deployment, as both a metadata storage and result storage backend.
values.yaml file.values.yaml file. These configurations enable the use of manual connection strings and provide an example of the database connection string format, which you can later update for your specific connection.Substitute astro-db-name with your own database name, if you need to create a new database.
Substitute astro-user-name and astro-user-password with your information. You can use an existing database for this step.
USAGE and CREATE privileges on the public schema to astro-user-name:Now, go into the database you created, which is astro-db-name in this example, and run the following queries
Retrieve the connection information for your external database. For example, with AWS, you can retrieve your endpoint information by Finding the connection information for an RDS for MySQL DB instance.
You need connection strings that define how Astronomer Software configures the connection to your external databases from your Airflow Deployment. The values of these strings are used when you define your metadataConnection or resultBackendConnection when you create, update, or upsert your Deployment.
Use the values for your astro-user-name, astro-user-password, astro-db-name, and the host information you retrieved to compose the connection strings in the following format, depending on whether you want to define a result backend connection or a metadata database connection.
metadataConnection:
resultBackendConnection:
The connection string format validation regex don’t cover the resultbackend connection string format, which includes db+. This is specifically required for the Celery executor worker. If the connection string doesn’t include db+, then Celery worker pod fails. The regex validation is not implemented because it adds the complications on format validation logic in different scenarios.
If you have PGBouncer enabled, and are using Postgres, you must configure metadataConnectionJson and resultBackendConnectionJson instead.
Use the values for your astro-user-name, astro-user-password, astro-db-name, and the host information you retrieved to compose the connection strings in the following format, depending on whether you want to define a result backend connection or a metadata database connection.
metadataConnectionJson:
resultBackendConnectionJson:
Use the Houston API to create your Deployment configuration.
The following example shows the mutation and queries for using createDeployment. See Houston API code examples for examples on how to use the update and upsert options for configuring your Deployment.