Warehouse connections

PreviewThis feature is in Preview.

A warehouse connection gives Astro Observe read access to your Snowflake or Databricks account. Observe uses the connection to discover the tables in your data warehouse and show them in the asset catalog.

Configure permissions

Before connecting to Astro Observe, configure the necessary permissions for your data platform.

Snowflake
Databricks

For Snowflake connections, the Observe role must have access to both the ACCOUNT_USAGE and INFORMATION_SCHEMA system tables. The service user must have a default warehouse configured to support table discovery.

Log into Snowflake

Log into Snowflake using a high-privilege role such as ACCOUNTADMIN.

Create a dedicated role for Observe

Create Observe role

CREATE ROLE IF NOT EXISTS ASTRO_OBSERVE_ROLE;

Create a read-only service user

Create a service user that Observe will use.

Create Observe service user

CREATE USER IF NOT EXISTS ASTRO_OBSERVE_USER
  DEFAULT_ROLE = ASTRO_OBSERVE_ROLE
  TYPE = SERVICE;

Assign the role to the user

Grant role to user

GRANT ROLE ASTRO_OBSERVE_ROLE TO USER ASTRO_OBSERVE_USER;

Grant the role the privileges Observe requires.

Replace YOUR_DB and other example names to match your Snowflake environment.

Example privileges for ASTRO_OBSERVE_ROLE

-- Grant warehouse access (replace COMPUTE_WH with your warehouse)
GRANT USAGE ON WAREHOUSE "COMPUTE_WH" TO ROLE ASTRO_OBSERVE_ROLE;
ALTER USER ASTRO_OBSERVE_USER SET DEFAULT_WAREHOUSE = 'COMPUTE_WH';

-- Metadata and object access:
GRANT USAGE, MONITOR ON DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
GRANT USAGE, MONITOR ON ALL SCHEMAS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

-- Read access:
GRANT SELECT ON ALL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
GRANT SELECT ON ALL VIEWS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
GRANT SELECT ON ALL EXTERNAL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

-- Optional: access to Snowflake usage views:
GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;
GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.TABLE_DML_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;

-- Or use:
-- GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ASTRO_OBSERVE_ROLE;

These are example grants. Replace database, schema, and warehouse names with values appropriate for your account and security policies.

Setup key-pair authentication in Snowflake (recommended)

Astronomer recommends key-pair authentication for Snowflake service users. Generate an RSA key pair, then assign the public key to the Observe service user to enable secure authentication.

Generate a password-protected private key and a public key

Run the following commands on a secure host to create an encrypted private key and a public key:

Generate RSA key pair (example)

# Create encrypted private key
openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8

# Create public key
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

Validate the key

openssl pkey -in rsa_key.p8 -check -noout

Assign the public key to the Snowflake user

Remove the -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY----- headers and newlines so the key is a compact single string. Assign the cleaned public key to the service user:

Assign public key to Snowflake user

ALTER USER ASTRO_OBSERVE_USER SET RSA_PUBLIC_KEY = '<your_cleaned_public_key>';

All Snowflake integrations require that the Observe role has access to both ACCOUNT_USAGE and INFORMATION_SCHEMA system tables. The service user must have a default warehouse configured for all discovery operations.

Observe’s Databricks Connector requires Unity Catalog enabled in your Databricks environment.

Get your Metastore ID

Get your Metastore ID using the Databricks CLI.

Get Metastore ID

databricks metastores current --profile <DATABRICKS_CONFIG_PROFILE>

List available schemas (optional)

List available system schemas to see what’s available.

List available schemas

databricks system-schemas list <METASTORE_ID>

Enable the required system schemas

Unity Catalog system tables provide metadata and logging information. These must be explicitly enabled before Observe can read from them.

Enable system schemas

databricks system-schemas enable <METASTORE_ID> query

Grant permissions to the service principal

Grant the necessary read permissions to the Observe Service Principal. Replace <astro_observe_service_principal> with the actual name of your service principal.

Grant permissions to service principal

GRANT USE SCHEMA ON system;

GRANT SELECT ON system.access.table_lineage TO <astro_observe_service_principal>;
GRANT SELECT ON system.access.column_lineage TO <astro_observe_service_principal>;
GRANT SELECT ON system.query.history TO <astro_observe_service_principal>;
GRANT SELECT ON system.lakeflow.jobs TO <astro_observe_service_principal>;
GRANT SELECT ON system.lakeflow.job_tasks TO <astro_observe_service_principal>;
GRANT SELECT ON system.lakeflow.job_run_timeline TO <astro_observe_service_principal>;
GRANT SELECT ON system.lakeflow.job_task_run_timeline TO <astro_observe_service_principal>;

These steps must be repeated per workspace that uses Unity Catalog
Make sure the Databricks CLI is configured (via profile or access token)
The above SQL commands must be run by a user with Unity Catalog admin privileges

Set up a connection

After you configure permissions for your data platform, create an Observe connection.

Open Connections

In the Astro UI, click Settings, then click Monitor Connections in the Observe section. Click New Connection (in the legacy UI, go to Observe > Connections and click New Connection).

Fill in connection details

Snowflake
Databricks

Complete the following fields:

Name: A name for the connection.
Description: Optional description.
Connection Type: Select Snowflake.
Polling Schedule: How frequently Observe polls Snowflake for metrics (examples: every 1 hour, 6 hours, 1 day). Polling frequency is the maximum rate at which Observe updates table metadata. More frequent polling can increase Snowflake compute costs.
Account Identifier: Your Snowflake account identifier (for example, FY02423-GP2141). Observe maps assets to a connection by account identifier.
Username: The Snowflake service user (ASTRO_OBSERVE_USER).
Private Key: Paste your private key for key-pair authentication if using key-pair auth.
Private Key Passphrase: Optional. The passphrase for an encrypted private key.

Only one Observe connection is allowed per Snowflake account identifier. If you have multiple Snowflake accounts, create a separate connection for each account identifier.

Save and start discovery

Click Create Connection. Observe begins the metadata extraction process and will discover your data assets and surface discovered tables in the Asset Catalog.

View warehouse tables in the asset catalog

Navigate to Catalog (Asset Catalog in the legacy UI), filter by your data platform (for example, Snowflake tables or Databricks tables), and select the desired table.

You can sort tables by popularity to quickly identify frequently used tables. Popularity rankings are based on query frequency and the number of unique users accessing each table.

Schema

The Schema tab shows table structure details:

Column names
Data types
Descriptions
Nullability
Default values

Overview

Get started

Develop Dags

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Warehouse connections

Configure permissions

Setup key-pair authentication in Snowflake (recommended)

Set up a connection

View warehouse tables in the asset catalog

Schema

​Configure permissions

​Setup key-pair authentication in Snowflake (recommended)

​Set up a connection

​View warehouse tables in the asset catalog

​Schema

Configure permissions

Setup key-pair authentication in Snowflake (recommended)

Set up a connection

View warehouse tables in the asset catalog

Schema