Grant Astronomer Access to your VPC
On Astronomer, all deployments that live in our Astronomer Cloud cluster route traffic through the same single NAT. In other words, we have 1 NAT gateway out of our VPC through which all internet-bound traffic goes through.
If you're looking to give Astronomer access to a database or a warehouse, follow the guidelines below. We'll use Amazon Redshift as an example.
Whitelist the Astronomer Cloud IP
To give Astronomer Cloud access to any database withiny your VPC, whitelist the following Static IP:
Whitelist Astronomer Cloud on AWS Redshift
Many of our customers choose Amazon Redshift as their Data Warehouse of choice.
Read below for a walkthrough of how to whitelist Astronomer Cloud on AWS Redshift.
Make your Redshift Cluster Publicly Accessible
If you didn’t do this on setup, it’s easy to modify.
- Go into the Redshift section of your AWS Console
- Choose the relevant Cluster
- Click “Modify Cluster"
- Toggle the “Publicly Accessible” option to “Yes”
- Click "Modify"
Whitelist the Cloud IP Address on AWS
Even though you’ve setup your Redshift to be publicly accessible, you’ll still want to limit where statements can be executed from.
With Astronomer, all queries will come from the same IP address:
Access VPC Security Groups
To whitelist this IP on your Cluster, go to “Security” on your Console and, depending on the specifics of your AWS account, click on “Go to the EC2 Console.”
Edit Inbound Rules
From there, click into the “Inbound” section of the relevant Security Group (which can be confirmed in the Cluster Profile page you were previously on in the “VPC security groups” section).
- Open up the Inbound rules by clicking “Edit”
- Add the Cloud IP address
- Click Save
Give your cluster a minute to update and then test access from within any Airflow deployment.
Add and Test the Connection
Because Redshift uses the same drivers as Postgres,you can add a connection to Airflow using the same methods as any other Postgres db.
Add a Connection
From the Airflow UI, go to Admin > Connections > "Create"
Pick a recognizable Conn Id (anything that will help you remember):
Postgresas the Conn Type
- Add in the endpoint that was generated for you when you created the cluster as the Host
Schemais the value of
Cluster Database Propertiessection of your Redshift cluster configuration
- Add in the username and password for whatever user you want to execute the queries
- Set the port to 5439 (not 5432)
Run a Query
After saving your connection:
- Go to Data Profiling>Ad Hoc Query from the top menu bar in the Airflow UI
- Choose the Redshift connection you just created
- Run a simple query
IF you're able to succesfully query, you're all done!