> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Data quality

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Astro Observe data quality helps you monitor tables to ensure data accuracy, completeness, and integrity across your pipelines. It automatically tracks key metrics such as column null percentages, schema changes, and table row counts to detect anomalies or unexpected shifts in your data.

## Configure permissions

Before connecting to Astro Observe, configure the necessary permissions for your data platform.

<Tabs>
  <Tab title="Snowflake">
    For Snowflake connections, the Observe role must have access to both the `ACCOUNT_USAGE` and `INFORMATION_SCHEMA` system tables. The service user must have a default warehouse configured to support discovery and ongoing data quality monitoring.

    <Steps>
      <Step title="Log into Snowflake">
        Log into Snowflake using a high-privilege role such as `ACCOUNTADMIN`.
      </Step>

      <Step title="Create a dedicated role for Observe">
        ```sql title="Create Observe role" wrap theme={null}
        CREATE ROLE IF NOT EXISTS ASTRO_OBSERVE_ROLE;
        ```
      </Step>

      <Step title="Create a read-only service user">
        Create a service user that Observe will use.

        ```sql title="Create Observe service user" wrap theme={null}
        CREATE USER IF NOT EXISTS ASTRO_OBSERVE_USER
          DEFAULT_ROLE = ASTRO_OBSERVE_ROLE
          TYPE = SERVICE;
        ```
      </Step>

      <Step title="Assign the role to the user">
        ```sql title="Grant role to user" wrap theme={null}
        GRANT ROLE ASTRO_OBSERVE_ROLE TO USER ASTRO_OBSERVE_USER;
        ```
      </Step>

      <Step title="Grant the role the privileges Observe requires.">
        Replace `YOUR_DB` and other example names to match your Snowflake environment.

        ```sql title="Example privileges for ASTRO_OBSERVE_ROLE" wrap theme={null}
        -- Grant warehouse access (replace COMPUTE_WH with your warehouse)
        GRANT USAGE ON WAREHOUSE "COMPUTE_WH" TO ROLE ASTRO_OBSERVE_ROLE;
        ALTER USER ASTRO_OBSERVE_USER SET DEFAULT_WAREHOUSE = 'COMPUTE_WH';

        -- Metadata and object access:
        GRANT USAGE, MONITOR ON DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT USAGE, MONITOR ON ALL SCHEMAS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

        -- Read access:
        GRANT SELECT ON ALL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON ALL VIEWS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON ALL EXTERNAL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

        -- Optional: access to Snowflake usage views:
        GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.TABLE_DML_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;

        -- Or use:
        -- GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ASTRO_OBSERVE_ROLE;
        ```

        <Note>These are example grants. Replace database, schema, and warehouse names with values appropriate for your account and security policies.</Note>
      </Step>
    </Steps>

    ## Setup key-pair authentication in Snowflake (recommended)

    Astronomer recommends key-pair authentication for Snowflake service users. Generate an RSA key pair, then assign the public key to the Observe service user to enable secure authentication.

    <Steps>
      <Step title="Generate a password-protected private key and a public key">
        Run the following commands on a secure host to create an encrypted private key and a public key:

        ```sh title="Generate RSA key pair (example)" wrap theme={null}
        # Create encrypted private key
        openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8

        # Create public key
        openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
        ```
      </Step>

      <Step title="Validate the key">
        ```sh wrap theme={null}
        openssl pkey -in rsa_key.p8 -check -noout
        ```
      </Step>

      <Step title="Assign the public key to the Snowflake user">
        Remove the `-----BEGIN PUBLIC KEY-----` and `-----END PUBLIC KEY-----` headers and newlines so the key is a compact single string. Assign the cleaned public key to the service user:

        ```sql title="Assign public key to Snowflake user" wrap theme={null}
        ALTER USER ASTRO_OBSERVE_USER SET RSA_PUBLIC_KEY = '<your_cleaned_public_key>';
        ```
      </Step>
    </Steps>

    All Snowflake integrations require that the Observe role has access to both `ACCOUNT_USAGE` and `INFORMATION_SCHEMA` system tables. The service user must have a default warehouse configured for all discovery and monitoring operations.
  </Tab>

  <Tab title="Databricks">
    Observe's Databricks Connector requires [Unity Catalog enabled](https://docs.databricks.com/aws/en/admin/system-tables#enable-system-tables) in your Databricks environment.

    <Steps>
      <Step title="Get your Metastore ID">
        Get your Metastore ID using the Databricks CLI.

        ```sh title="Get Metastore ID" wrap theme={null}
        databricks metastores current --profile <DATABRICKS_CONFIG_PROFILE>
        ```
      </Step>

      <Step title="List available schemas (optional)">
        List available system schemas to see what's available.

        ```sh title="List available schemas" wrap theme={null}
        databricks system-schemas list <METASTORE_ID>
        ```
      </Step>

      <Step title="Enable the required system schemas">
        Unity Catalog system tables provide metadata and logging information. These must be explicitly enabled before Observe can read from them.

        ```sh title="Enable system schemas" wrap theme={null}
        databricks system-schemas enable <METASTORE_ID> query
        ```
      </Step>

      <Step title="Grant permissions to the service principal">
        Grant the necessary read permissions to the Observe Service Principal. Replace `<astro_observe_service_principal>` with the actual name of your service principal.

        ```sql title="Grant permissions to service principal" wrap theme={null}
        GRANT USE SCHEMA ON system;

        GRANT SELECT ON system.access.table_lineage TO <astro_observe_service_principal>;
        GRANT SELECT ON system.access.column_lineage TO <astro_observe_service_principal>;
        GRANT SELECT ON system.query.history TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.jobs TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_tasks TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_run_timeline TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_task_run_timeline TO <astro_observe_service_principal>;
        ```

        <Note>
          * These steps must be repeated per workspace that uses Unity Catalog
          * Make sure the Databricks CLI is configured (via profile or access token)
          * The above SQL commands must be run by a user with Unity Catalog admin privileges
        </Note>
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Set up a connection

After you configure permissions for your data platform, create an Observe connection.

<Steps>
  <Step title="Open Connections">
    In the Astro UI main navigation, go to **Dashboards** > **Data Quality** and click **+ New Connection** (in the legacy UI, navigate to **Data Quality** > **Connections** and click **+ Connection**).
  </Step>

  <Step title="Fill in connection details">
    <Tabs>
      <Tab title="Snowflake">
        Complete the following fields:

        * **Name:** A name for the connection.
        * **Description:** Optional description.
        * **Connection Type:** Select **Snowflake**.
        * **Polling Schedule:** How frequently Observe polls Snowflake for metrics (examples: every 1 hour, 6 hours, 1 day). Polling frequency is the maximum rate at which Observe updates data quality metrics and monitors; more frequent polling may increase Snowflake compute costs.
        * **Account Identifier:** Your Snowflake account identifier (for example, `FY02423-GP2141`). Observe maps assets to a connection by account identifier.
        * **Username:** The Snowflake service user (`ASTRO_OBSERVE_USER`).
        * **Private Key:** Paste your private key for key-pair authentication if using key-pair auth.

        <Note>
          Only one Observe connection is allowed per Snowflake account identifier. If you have multiple Snowflake accounts, create a separate connection for each account identifier.
        </Note>
      </Tab>

      <Tab title="Databricks">
        Complete the following fields:

        * **Name:** A name for the connection.
        * **Description:** Optional description to help identify the purpose of the connection.
        * **Connection Type:** Select **Databricks**.
        * **Polling Schedule:** How frequently Observe polls Databricks for metrics (examples: every 1 hour, 6 hours, 1 day). Polling frequency is the maximum rate at which Observe updates data quality metrics and monitors; more frequent polling may increase Databricks compute costs.
        * **Host:** Enter the server hostname for your Databricks warehouse.
        * **HTTP Path:** Enter the HTTP Path for your Databricks warehouse.
        * **Password:** This can be either a personal access token or an OAuth secret from a service principal. Astronomer recommends [creating a service principal and using it to generate an OAuth secret](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m#prerequisite-create-a-service-principal).

        <Note>
          Your Databricks role must have Account Admin or Workspace Admin permissions in order to generate a Service Principal and secret.
        </Note>

        <Info>
          To find **Host** and **HTTP Path** for your Databricks warehouse, navigate to **SQL Warehouses** in Databricks, select the warehouse you want to connect, and click **Connection details**. Here you will find Host and HTTP Path.
        </Info>
      </Tab>
    </Tabs>
  </Step>

  <Step title="Save and start discovery">
    Click **Create**. Observe begins the metadata extraction process and will discover your data assets and surface discovered tables in the Asset Catalog.
  </Step>
</Steps>

## Navigating data quality in Astro Observe

### Asset Catalog

Navigate to **Catalog** (**Asset Catalog** in the legacy UI), filter by your data platform (for example, **Snowflake tables** or **Databricks tables**), and select the desired table.

<Info>
  You can sort tables by *popularity* to quickly identify frequently used tables. Popularity rankings are based on query frequency and the number of unique users accessing each table.
</Info>

### Schema

The **Schema** tab shows table structure details:

* Column names
* Data types
* Completeness status
* Nullability
* Default values

You can enable monitoring for specific columns to actively track completeness.

### Event Timeline

The **Event Timeline** tab shows data quality events for a selected timeframe. Events are color-coded by severity: **Success**, **Neutral**, and **Failure**. Click an event to view details, historical patterns, and affected metrics.

### Data quality

The **data quality** tab provides visualizations for monitored metrics:

* **Table Volume:** track changes in row counts and percent change over time to identify unexpected fluctuations.
* **Completeness:** visualize column null percentages against thresholds to surface completeness problems.

### Monitors

To create and manage data quality monitors, see [Monitors in Astro Observe](/docs/astro/observe-monitors#data-quality-monitors).

### Triggered monitor overview

To see a high-level overview of your organization's data quality, click **Data Quality** in the navigation. Here you can see a summary of triggered data quality monitors from the last week or month, grouped by severity and check type.

Click any triggered monitor to investigate it and see the underlying data that triggered the monitor's conditions.

<Frame>
  <img src="https://mintcdn.com/astronomer/Ct4jHNJYwyny9aUD/images/docs/dq-overview.png?fit=max&auto=format&n=Ct4jHNJYwyny9aUD&q=85&s=5671ee2c23d09367e2b56454f15697b5" alt="Data quality issues overview dashboard" width="2970" height="1630" data-path="images/docs/dq-overview.png" />
</Frame>