> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up APC audit log shipping

This document explains how to enable the Astro Private Cloud (APC) audit log sidecar and ship events to one supported sink. For background on what the feature does and which configurations are supported, see [APC audit logging overview](/docs/astro-private-cloud/v-2-x/audit-logging-overview).

<Note>
  Exactly one sink can be enabled per installation for this release. Enabling the sidecar with zero or more than one sink causes a Helm validation error.
</Note>

## Prerequisites

* An Astro Private Cloud 2.x installation that you can upgrade with Helm.
* Access to the Helm values file used by the installation.
* `kubectl` configured against the target cluster.
* Permissions to create or modify cloud resources for the sink you choose:
  * CloudWatch on EKS: permissions to create IAM policies and roles, and a CloudWatch log group.
  * GCP Cloud Logging on GKE: permissions to create Google service accounts and grant IAM bindings in the target project.
  * Elasticsearch: a reachable endpoint and, if required, credentials and a CA certificate.

### Choose a sink

Use the AWS CloudWatch Logs sink when the Astro Private Cloud control plane runs on Amazon EKS. Use the GCP Cloud Logging sink when it runs on Google Kubernetes Engine (GKE). Use the Elasticsearch sink on Amazon EKS, GKE, or Azure Kubernetes Service (AKS).

<Tabs>
  <Tab title="AWS CloudWatch Logs">
    Use this sink when the Astro Private Cloud control plane runs on Amazon EKS. The recommended authentication method is IAM Roles for Service Accounts (IRSA). Static AWS credentials held in a Kubernetes secret are supported as a fallback when IRSA isn't in use on the EKS cluster.

    #### Prerequisites

    * The Astro Private Cloud control plane runs on Amazon EKS.
    * The AWS CLI is installed and authenticated against the target account.
    * For the IRSA path, the EKS cluster has, or can be associated with, an OIDC identity provider.
    * For the static-credentials path, an IAM principal with permission to write to the target CloudWatch log group.

    ##### Environment variables

    The following variables are referenced throughout this section. Set them to match your installation before running the commands.

    ```bash wrap theme={null}
    export EKS_CLUSTER_NAME="<cluster-name>"
    export AWS_REGION="<region>"
    export AWS_ACCOUNT_ID="<account-id>"
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
    export IRSA_ROLE_NAME="HoustonCloudWatchRole"
    export CW_LOG_GROUP="/astronomer/houston/audit"
    ```

    ##### Configure IRSA (recommended)

    <Steps>
      <Step title="Create the CloudWatch log group">
        ```bash wrap theme={null}
        aws logs create-log-group \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION"
        ```

        Optionally set a retention policy on the log group:

        ```bash wrap theme={null}
        aws logs put-retention-policy \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION" \
          --retention-in-days 30
        ```
      </Step>

      <Step title="Associate the EKS cluster OIDC provider">
        ```bash wrap theme={null}
        OIDC_URL=$(aws eks describe-cluster \
          --name "$EKS_CLUSTER_NAME" \
          --region "$AWS_REGION" \
          --query "cluster.identity.oidc.issuer" \
          --output text)
        OIDC_ID=${OIDC_URL#https://}

        aws iam list-open-id-connect-providers \
          | grep "$(echo "$OIDC_ID" | awk -F/ '{print $NF}')" \
          || eksctl utils associate-iam-oidc-provider \
               --cluster "$EKS_CLUSTER_NAME" \
               --region "$AWS_REGION" \
               --approve
        ```
      </Step>

      <Step title="Create the IAM policy">
        ```bash expandable wrap theme={null}
        cat > /tmp/houston-cloudwatch-policy.json <<EOF
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "CloudWatchDescribe",
              "Effect": "Allow",
              "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
              ],
              "Resource": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:*"
            },
            {
              "Sid": "CloudWatchWriteAuditGroup",
              "Effect": "Allow",
              "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": [
                "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}",
                "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}:*"
              ]
            }
          ]
        }
        EOF

        aws iam create-policy \
          --policy-name HoustonCloudWatchLogsPolicy \
          --policy-document file:///tmp/houston-cloudwatch-policy.json
        ```
      </Step>

      <Step title="Create the IRSA role and attach the policy">
        ```bash wrap theme={null}
        cat > /tmp/trust-policy.json <<EOF
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ID}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "${OIDC_ID}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA}",
                  "${OIDC_ID}:aud": "sts.amazonaws.com"
                }
              }
            }
          ]
        }
        EOF

        aws iam create-role \
          --role-name "$IRSA_ROLE_NAME" \
          --assume-role-policy-document file:///tmp/trust-policy.json

        aws iam attach-role-policy \
          --role-name "$IRSA_ROLE_NAME" \
          --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/HoustonCloudWatchLogsPolicy"
        ```

        If the IRSA role already exists and you only need to bind it to a new EKS cluster, use `aws iam update-assume-role-policy` against the existing role instead of `aws iam create-role`.
      </Step>

      <Step title="Apply the Helm values override">
        Add the following to the values file used by the installation and run `helm upgrade`:

        ```yaml wrap theme={null}
        astronomer:
          houston:
            serviceAccount:
              annotations:
                eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/HoustonCloudWatchRole"
            logging:
              loggingSidecar:
                enabled: true
                cloudwatch:
                  enabled: true
                  region: "<AWS_REGION>"
                  logGroupName: "/astronomer/houston/audit"
                  useIRSA: true
        ```
      </Step>
    </Steps>

    #### Configure static AWS credentials (fallback)

    Use this configuration on EKS when IRSA isn't in use.

    <Steps>
      <Step title="Create the CloudWatch log group">
        ```bash wrap theme={null}
        aws logs create-log-group \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION"
        ```
      </Step>

      <Step title="Create a Kubernetes secret with AWS credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-cloudwatch-creds \
          --from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
          --from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>" \
          -n "$K8S_NAMESPACE"
        ```

        The IAM principal whose credentials you use must be allowed to write to the target log group. The policy shown in the IRSA section is a suitable template.
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                cloudwatch:
                  enabled: true
                  region: "<AWS_REGION>"
                  logGroupName: "/astronomer/houston/audit"
                  useIRSA: false
                  secretName: "houston-cloudwatch-creds"
        ```
      </Step>
    </Steps>

    #### Verify

    After the upgrade completes, confirm that the Vector sidecar is running and that audit events are reaching CloudWatch:

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

    aws logs tail "$CW_LOG_GROUP" --region "$AWS_REGION" --since 5m
    ```

    Perform any action that the APC API audits, such as creating a Workspace, and confirm a matching event appears in the log group within a few seconds.
  </Tab>

  <Tab title="GCP Cloud Logging">
    Use this sink when the Astro Private Cloud control plane runs on Google Kubernetes Engine (GKE). The recommended authentication method is Workload Identity. A GCP service account JSON key held in a Kubernetes secret is supported as a fallback when Workload Identity isn't in use on the GKE cluster.

    #### Prerequisites

    * The Astro Private Cloud control plane runs on GKE.
    * `gcloud` is installed and authenticated against the target Google Cloud project.
    * For the Workload Identity path, the GKE cluster has Workload Identity enabled.
    * For the service-account-key path, a Google service account with the `roles/logging.logWriter` role in the target Google Cloud project.

    ##### Environment variables

    ```bash wrap theme={null}
    export GCP_PROJECT_ID="<project-id>"
    export GKE_CLUSTER_NAME="<cluster-name>"
    export GKE_LOCATION="<zone-or-region>"
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
    export GCP_SA_NAME="houston-logging"
    export GCP_SA_EMAIL="${GCP_SA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"
    ```

    ##### Configure Workload Identity (recommended)

    <Steps>
      <Step title="Verify Workload Identity is enabled">
        ```bash wrap theme={null}
        gcloud container clusters describe "$GKE_CLUSTER_NAME" \
          --location "$GKE_LOCATION" \
          --format="value(workloadIdentityConfig.workloadPool)"
        ```

        Expected output is `${GCP_PROJECT_ID}.svc.id.goog`. If the output is empty, enable Workload Identity on the cluster before continuing.
      </Step>

      <Step title="Create the Google service account and grant log write access">
        ```bash wrap theme={null}
        gcloud iam service-accounts create "$GCP_SA_NAME" \
          --display-name="Houston Audit Log Writer" \
          --project="$GCP_PROJECT_ID"

        gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
          --member="serviceAccount:${GCP_SA_EMAIL}" \
          --role="roles/logging.logWriter"
        ```
      </Step>

      <Step title="Bind the Google service account to the APC API Kubernetes service account">
        ```bash wrap theme={null}
        gcloud iam service-accounts add-iam-policy-binding "$GCP_SA_EMAIL" \
          --role="roles/iam.workloadIdentityUser" \
          --member="serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${K8S_SA}]" \
          --project="$GCP_PROJECT_ID"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            serviceAccount:
              annotations:
                iam.gke.io/gcp-service-account: "<GCP_SA_EMAIL>"
            logging:
              loggingSidecar:
                enabled: true
                gcpCloudLogging:
                  enabled: true
                  projectId: "<GCP_PROJECT_ID>"
                  logId: "houston-audit"
                  resource:
                    type: "k8s_container"
                    location: "<GKE_LOCATION>"
                    clusterName: "<GKE_CLUSTER_NAME>"
                  severityKey: "level"
                  useWorkloadIdentity: true
        ```

        `projectId`, `resource.location`, and `resource.clusterName` are required when `gcpCloudLogging.enabled` is true. Leaving any of these empty causes a Helm validation error.
      </Step>
    </Steps>

    #### Configure a service account key (fallback)

    Use this configuration on GKE when Workload Identity isn't in use.

    <Steps>
      <Step title="Create a key for the Google service account">
        ```bash wrap theme={null}
        gcloud iam service-accounts keys create /tmp/houston-logging-key.json \
          --iam-account="$GCP_SA_EMAIL"
        ```
      </Step>

      <Step title="Create a Kubernetes secret from the key">
        ```bash wrap theme={null}
        kubectl create secret generic houston-gcp-logging-creds \
          --from-file=key.json=/tmp/houston-logging-key.json \
          --namespace "$K8S_NAMESPACE"

        rm /tmp/houston-logging-key.json
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                gcpCloudLogging:
                  enabled: true
                  projectId: "<GCP_PROJECT_ID>"
                  logId: "houston-audit"
                  resource:
                    type: "k8s_container"
                    location: "<GKE_LOCATION>"
                    clusterName: "<GKE_CLUSTER_NAME>"
                  severityKey: "level"
                  useWorkloadIdentity: false
                  credentialsSecretName: "houston-gcp-logging-creds"
                  credentialsSecretKey: "key.json"
        ```
      </Step>
    </Steps>

    #### Verify

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

    gcloud logging read \
      "logName=\"projects/${GCP_PROJECT_ID}/logs/houston-audit\"" \
      --project="$GCP_PROJECT_ID" \
      --limit=5 \
      --freshness=5m
    ```

    Perform any action that the APC API audits and confirm a matching entry appears in the log stream within a few seconds.
  </Tab>

  <Tab title="Elasticsearch">
    Use this sink to ship APC API audit events to an external Elasticsearch cluster. The Elasticsearch sink is supported when the Astro Private Cloud control plane runs on Amazon EKS, GKE, or Azure Kubernetes Service (AKS). This sink needs no cloud-provider IAM configuration.

    #### Prerequisites

    * A reachable Elasticsearch endpoint on version 8.
    * For basic auth, a username and password.
    * For custom CA trust, a PEM-formatted CA certificate.

    ##### Environment variables

    ```bash wrap theme={null}
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export ES_ENDPOINT="https://es.example.com:9200"
    export ES_USERNAME="elastic"
    export ES_PASSWORD="<password>"
    ```

    #### Configure basic auth

    <Steps>
      <Step title="Create a Kubernetes secret with the Elasticsearch credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-elasticsearch-creds \
          --from-literal=username="$ES_USERNAME" \
          --from-literal=password="$ES_PASSWORD" \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "basic"
                    secretName: "houston-elasticsearch-creds"
        ```
      </Step>
    </Steps>

    #### Configure basic auth with a custom CA

    Use this variant when the Elasticsearch endpoint presents a certificate signed by a private or internal CA that isn't already trusted by the Vector sidecar.

    <Steps>
      <Step title="Create a Kubernetes secret with the credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-elasticsearch-creds \
          --from-literal=username="$ES_USERNAME" \
          --from-literal=password="$ES_PASSWORD" \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Create a Kubernetes secret with the CA certificate">
        The secret key must be named `ca.pem`. The Vector sidecar expects the CA at `/etc/es-tls/ca.pem` and the chart mounts the referenced secret to that path.

        ```bash wrap theme={null}
        kubectl create secret generic houston-es-ca \
          --from-file=ca.pem=/path/to/ca-certificate.pem \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "basic"
                    secretName: "houston-elasticsearch-creds"
                  tls:
                    enabled: true
                    caSecretName: "houston-es-ca"
        ```
      </Step>
    </Steps>

    #### Configure anonymous auth

    Use this configuration when the Elasticsearch endpoint accepts unauthenticated requests. This configuration needs no credentials secret.

    ```yaml wrap theme={null}
    astronomer:
      houston:
        logging:
          loggingSidecar:
            enabled: true
            elasticsearch:
              enabled: true
              endpoint: "http://es.example.com:9200"
              index: "houston-audit-%Y.%m.%d"
              apiVersion: "v8"
              auth:
                strategy: "none"
    ```

    ##### Configure anonymous auth with a custom CA

    Use this variant when the Elasticsearch endpoint accepts unauthenticated requests over TLS and presents a certificate signed by a private or internal CA.

    <Steps>
      <Step title="Create a Kubernetes secret with the CA certificate">
        The secret key must be named `ca.pem`. The Vector sidecar expects the CA at `/etc/es-tls/ca.pem` and the chart mounts the referenced secret to that path.

        ```bash wrap theme={null}
        kubectl create secret generic houston-es-ca \
          --from-file=ca.pem=/path/to/ca-certificate.pem \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "none"
                  tls:
                    enabled: true
                    caSecretName: "houston-es-ca"
        ```
      </Step>
    </Steps>

    #### Verify

    Check the Vector sidecar logs for sink health:

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20
    ```

    Query the Elasticsearch endpoint for the `houston-audit-*` indices. If the endpoint requires authentication:

    ```bash wrap theme={null}
    curl --cacert /path/to/ca-certificate.pem -u "$ES_USERNAME:$ES_PASSWORD" \
      "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"
    ```

    If the endpoint accepts anonymous requests:

    ```bash wrap theme={null}
    curl --cacert /path/to/ca-certificate.pem \
      "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"
    ```

    Omit the `--cacert` flag when the endpoint uses a publicly trusted certificate. Perform any action that the APC API audits and confirm that a new document appears in the matching `houston-audit-*` index.
  </Tab>
</Tabs>

## Disable audit log shipping

To stop shipping audit events, set the sidecar to disabled and run `helm upgrade`:

```yaml wrap theme={null}
astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: false
```

`helm upgrade` removes the Vector sidecar from the next Pod restart. The APC API continues to emit audit events to standard output, so you can still inspect recent events with `kubectl logs` on the APC API and APC Worker Pods.

## Common issues

<AccordionGroup>
  <Accordion title="Helm reports 'supports exactly one sink at a time'">
    The sidecar is enabled with more than one sink. Edit the values file so that only one of `cloudwatch.enabled`, `gcpCloudLogging.enabled`, or `elasticsearch.enabled` is true, then run `helm upgrade` again.
  </Accordion>

  <Accordion title="Helm reports 'requires at least one supported sink'">
    The sidecar is enabled but no sink is selected. Set one of `cloudwatch.enabled`, `gcpCloudLogging.enabled`, or `elasticsearch.enabled` to true, or set `loggingSidecar.enabled` to false.
  </Accordion>

  <Accordion title="Helm reports a required GCP Cloud Logging value is missing">
    `gcpCloudLogging.projectId`, `gcpCloudLogging.resource.location`, and `gcpCloudLogging.resource.clusterName` are required when GCP Cloud Logging is enabled. Whitespace-only values are also rejected. Set all three to concrete values and run `helm upgrade` again.
  </Accordion>

  <Accordion title="Vector reports AccessDenied errors on CloudWatch">
    The IAM principal that Vector uses can't write to the target log group.

    * For IRSA, check that the IRSA role trust policy names the correct OIDC provider, namespace, and service account, and that `HoustonCloudWatchLogsPolicy` is attached to the role. Also verify that the `eks.amazonaws.com/role-arn` annotation on `houston.serviceAccount.annotations` points to the same role ARN.
    * For static credentials, check that `houston-cloudwatch-creds` contains valid `aws_access_key_id` and `aws_secret_access_key` values and that the corresponding IAM user or role is allowed to write to the target log group.
  </Accordion>

  <Accordion title="GCP Cloud Logging rejects entries with HTTP 400">
    The monitored resource is invalid. Confirm that `resource.type` is `k8s_container`, and that `resource.location` and `resource.clusterName` match the GKE cluster as it appears in Cloud Logging.
  </Accordion>

  <Accordion title="Vector can't connect to Elasticsearch">
    Check that `endpoint` is reachable from within the cluster and that, if TLS is in use, the CA certificate in `caSecretName` signs the server certificate presented by the endpoint. If basic auth is enabled, verify that the username and password in `houston-elasticsearch-creds` are correct.
  </Accordion>
</AccordionGroup>

## Next steps

* For a field-by-field description of each value under `houston.logging.loggingSidecar`, see [Audit logging configuration reference](/docs/astro-private-cloud/v-2-x/audit-logging-reference).
* For the shape of each audit event and the list of audited operations, see [Audit log schema and operations](/docs/astro-private-cloud/v-2-x/audit-log-schema).