> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure task log collection and exporting to Elasticsearch

Apache Airflow task logs are stored in a logging backend to ensure you can access them after your Pods terminate. By default, Astro Private Cloud uses [Vector](https://vector.dev/) to collect task logs and export them to an Elasticsearch instance.

You can configure how Astro Private Cloud collects Deployment task logs and exports them to Elasticsearch. The following are the supported methods for exporting task logs to Elasticsearch:

* Using a DaemonSet Pod on each Kubernetes node in your cluster.
* Using container sidecars for Deployment components.

## Export task logs using a Vector DaemonSet

<Warning>
  Exporting task logs using a Vector DaemonSet isn't supported for Airflow 3.
</Warning>

By default, Astro Private Cloud uses a Vector DaemonSet to aggregate task logs. This is the workflow for the default implementation:

* Deployments write task logs to `stdout`.
* Kubernetes takes the output from `stdout` and writes it to the Deployment’s node.
* A Vector Pod reads logs from the node and forwards them to Elasticsearch.

Astronomer recommends using Vector DaemonSet for organizations that:

* Run longer tasks using Celery executor.
* Run Astro Private Cloud in a dedicated cluster.
* Run privileged containers in a cluster with a ClusterRole.

This approach isn't suited for organizations that don't allow logging container to run in privileged mode and run many small tasks using the Kubernetes executor. Because task logs exist only for the lifetime of the Pod, your Pods running small tasks might complete before Vector can collect their task logs.

## Export logs using container sidecars

You can use a logging sidecar container to collect and export logs. In this implementation:

* Each container running an Airflow component for a Deployment receives its own [Vector](https://vector.dev/) sidecar.
* Task logs are written to a shared directory.
* The Vector sidecar reads logs from the shared directory and writes them to Elasticsearch.

This implementation is recommended for organizations that:

* Run Astro Private Cloud in a multi-tenant cluster, where security is a concern.
* Use the Kubernetes executor to run many short-lived tasks, which requires improved reliability.

### Configure logging sidecars

1. Retrieve your `values.yaml` file. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

2. Add the following entry to your `values.yaml` file:

   ```yaml wrap theme={null}
   global:
     daemonsetLogging:
       enabled: false
     logging:
       loggingSidecar:
         enabled: true
         name: sidecar-log-consumer
   ```

   <Tip>
     If you're migrating from Fluentd, you must also set the following configuration so that Astro Private Cloud can retain logs:

     ```yaml wrap theme={null}
     global:
       logging:
         indexNamePrefix: <your-index-prefix>
     ```
   </Tip>

3. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

#### Customize Vector logging sidecars

You can customize the default Astronomer Vector logging sidecar to have different transformations and sinks based on your team's requirements. This is useful if you want to annotate, customize, or filter your logs before sending them to your logging platform.

1. In the **Astro UI**, go to your **Clusters** page and select your cluster.

2. In the cluster details, click **Edit** in the **Deployment Configuration** section and add the following override in the **Configuration Override** field:

   ```yaml wrap theme={null}
   global:
     loggingSidecar:
       enabled: true
       name: sidecar-log-consumer
       customConfig: true
   ```

   For more information on using the **Configuration Override** in the UI, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

<Note>
  Pushing this change updates the configuration for the cluster. Individual Deployments will receive the new sidecar logging configuration once they are redeployed.
</Note>

4. Create a custom [vector configuration `yaml` file](https://vector.dev/docs/reference/configuration/) to change how and where sidecars forward your logs. The following examples are template configurations for each commonly used external logging service. For the complete default logging sidecar configmap, see the [Astronomer GitHub](https://github.com/astronomer/airflow-chart/blob/master/templates/logging-sidecar-configmap.yaml).

<Tabs>
  <Tab title="Elasticsearch">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_airflow_logs:
        type: remap
        inputs:
          - airflow_log_files
        source: |
          .component = "${COMPONENT:--}"
          .workspace = "${WORKSPACE:--}"
          .release = "${RELEASE:--}"
          .date_nano = parse_timestamp!(.@timestamp, format: "%Y-%m-%dT%H:%M:%S.%f%Z")

      filter_common_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: '!includes(["worker","scheduler"], .component)'

      filter_scheduler_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["scheduler"], .component)'

      filter_worker_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["worker"], .component)'

      filter_gitsyncrelay_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["git-sync-relay"], .component)'

      transform_task_log:
        type: remap
        inputs:
          - filter_worker_logs
          - filter_scheduler_logs
        source: |-
          . = parse_json(.message) ?? .
          .@timestamp = parse_timestamp(.timestamp, "%Y-%m-%dT%H:%M:%S%Z") ?? now()
          .check_log_id = exists(.log_id)
          if .check_log_id != true {
          .log_id = join!([to_string!(.dag_id), to_string!(.task_id), to_string!(.execution_date), to_string!(.try_number)], "_")
          }
          .offset = to_int(now()) * 1000000000 + to_unix_timestamp(now()) * 1000000

      final_task_log:
        type: remap
        inputs:
          - transform_task_log
        source: |
          .component = "${COMPONENT:--}"
          .workspace = "${WORKSPACE:--}"
          .release = "${RELEASE:--}"
          .date_nano = parse_timestamp!(.@timestamp, format: "%Y-%m-%dT%H:%M:%S.%f%Z")

      transform_remove_fields:
        type: remap
        inputs:
          - final_task_log
          - filter_common_logs
          - filter_gitsyncrelay_logs
        source: |
          del(.host)
          del(.file)

    ## Configuration for ElasticSearch sink

    sinks:
      out:
        type: elasticsearch

    ## Specify the transforms you want to run before your logs are exported

        inputs:
          - transform_remove_fields
        mode: bulk
        compression: none
        endpoint: "http://example-host:<example-port>"
        auth:
          strategy: "basic"
          user: "example-user"
          password : "example-pass"
        bulk:
          index: "vector.${RELEASE:--}.%Y.%m.%d"
          action: create

    ```
  </Tab>

  <Tab title="Honeycomb">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_syslog:
        type: add_fields
        inputs:
          - generate_syslog
        fields:
          component: "${COMPONENT:--}"
          workspace: "${WORKSPACE:--}"
          release: "${RELEASE:--}"
      transform_task_log:
        type: remap
        inputs:
          - transform_syslog
        source: |-
          # Parse Syslog input. The "!" means that the script should abort on error.
          . = parse_json!(.message)
          .@timestamp = parse_timestamp(.timestamp, "%Y-%m-%dT%H:%M:%S%Z") ?? now()
          .check_log_id = exists(.log_id)
          if .check_log_id != true {
          .log_id = join!([.dag_id, .task_id, .execution_date, .try_number], "_")
          }
          .offset = to_int(now()) * 1000000000 + to_unix_timestamp(now()) * 1000000
    # Configuration for Datadog sinks
    sinks:
      my_sink_id:
        type: datadog_logs
        # Specify the transforms you want to run before your logs are exported.
        inputs:
          - transform_task_log
        site: us1.datadoghq.com
        default_api_key: <your-api-key>
        encoding:
          codec: json
    ```
  </Tab>

  <Tab title="Datadog">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_syslog:
        type: add_fields
        inputs:
          - generate_syslog
        fields:
          component: "${COMPONENT:--}"
          workspace: "${WORKSPACE:--}"
          release: "${RELEASE:--}"

    ## Configuration for Honeycomb sinks

    sinks:
      my_sink_id:
        type: honeycomb

    ## Specify the transforms you want to run before your logs are exported

        inputs:
          - transform_syslog
        api_key: <your-api-key>
        dataset: my-honeycomb-dataset

    ```
  </Tab>
</Tabs>

5. Run the following command to add the configuration file to your cluster as a Kubernetes secret:

   ```bash wrap theme={null}
   kubectl create secret generic sidecar-config --from-file=vector-values.yaml=vector-values.yaml
   ```

6. Run the following command to annotate the secret so that it's automatically applied to all new Deployments:

   ```bash wrap theme={null}
   kubectl annotate secret secret-name astronomer.io/commander-sync="platform-release=astronomer"
   ```

7. Run the following command to sync existing Deployments with the new configuration:

   ```bash wrap theme={null}
   kubectl create job --from=cronjob/astronomer-config-syncer sync-secrets -n astronomer
   ```

## Use an external Elasticsearch instance for Airflow task log management

Add Airflow task logs from your Astronomer Deployment to an existing Elasticsearch instance on [Elastic Cloud](https://www.elastic.co/cloud/) to centralize log management and analysis. Centralized log management allows you to quickly identify, troubleshoot, and resolve task failure issues. Although these examples use Elastic Cloud, you can also use AWS Managed OpenSearch Service or any other elastic service (managed or hosted). With an external Elasticsearch instance configured for Astro Private Cloud, you can see the logs in your Elasticsearch instance and browse the logs from the APC UI.

<Note>If you use an existing Elasticsearch instance, make sure that the index template is configured to enable auto creation of new indices.</Note>

### Create an Elastic Deployment and endpoint

1. In your browser, go to `https://cloud.elastic.co/` and create a new Elastic Cloud deployment. See [Create a deployment](https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html#ec-create-deployment).

2. Copy and save your Elastic Cloud deployment credentials when the **Save the deployment credentials** screen appears.

3. On the Elastic dashboard, click the **Gear** icon for your Deployment.

4. Click **Copy endpoint** next to **Elasticsearch**.

5. (Optional) Test the Elastic Cloud deployment endpoint:
   * Open a new browser window, paste the endpoint you copied in step 4 in the **Address** bar, and then press **Enter**.
   * Enter the username and password you copied in step 2 and click **Sign in**. Output similar to the following appears:
   ```text wrap theme={null}
       name	"instance-0000000000"
       cluster_name	"<cluster-name>"
       cluster_uuid	"<cluster-uuid>"
       version
       number	"8.3.2"
       build_type	"docker"
       build_hash	"8b0b1f23fbebecc3c88e4464319dea8989f374fd"
       build_date	"2022-07-06T15:15:15.901688194Z"
       build_snapshot	false
       lucene_version	"9.2.0"
       minimum_wire_compatibility_version	"7.17.0"
       minimum_index_compatibility_version	"7.0.0"
       tagline	"You Know, for Search"
   ```

### Save your Elastic Cloud deployment credentials

After you've created an Elastic deployment and endpoint, you have two options to store your Elastic deployment credentials. You can store the credentials in your Astro Private Cloud Helm values, or for greater security, as a secret in your Astro Private Cloud Kubernetes cluster. For additional information about adding an Astro Private Cloud configuration change, see [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

<Note>
  Updating the Elasticsearch host in your data plane `values.yaml` doesn't automatically update the cluster configuration database. After you upgrade your data plane Helm release, manually update the Elasticsearch proxy host through the UI:

  1. In the Astro UI, go to **Clusters** and select your cluster.

  2. Click **Edit** to unlock **Configuration Override**.

  3. Add the following to the **Configuration Override**:

     ```json wrap theme={null}
     {
       "helm": {
         "airflow": {
           "elasticsearch": {
             "enabled": true,
             "connection": {
               "host": "<your-elasticsearch-host>",
               "port": 9200
             }
           }
         }
       }
     }
     ```

  4. Click **Update cluster** to apply your changes.

  5. Update each existing Deployment to apply the new Elasticsearch configuration.

  For more information, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).
</Note>

<Tabs>
  <Tab title="values.yaml">
    1. Run the following command to base64 encode your Elastic Cloud deployment credentials:

    ```bash wrap theme={null}
       echo -n "<username>:<password>" | base64
    ```

    2. Add the following entry to your `values.yaml` file:

       ```yaml wrap theme={null}
       global:
       daemonsetLogging:
         enabled: true
       customLogging:
         enabled: true
         scheme: https
         # host endpoint copied from elasticsearch console with https
         # and port number removed.
         host: "<host-URL>"
         port: "9243"
         # encoded credentials from above step 1
         secret: "<encoded credentials>"
       ```

    3. Add the following entry to your `values.yaml` file to disable internal logging:

       ```yaml wrap theme={null}
       tags:
       logging: false
       ```

    4. Run the following command to upgrade the Astro Private Cloud release version in the `values.yaml` file:

       ```bash wrap theme={null}
       helm upgrade -f values.yaml --version=2.0.0 --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
       ```
  </Tab>

  <Tab title="Kubernetes secret">
    1. Run the following command to create a secret for your Elastic Cloud Deployment credentials in the Kubernetes cluster:

    ```bash wrap theme={null}
    kubectl create secret generic elasticcreds --from-literal elastic=<username>:<password> --namespace=<your-platform-namespace>
    ```

    2. Add the following entry to your `values.yaml` file:

       ```yaml wrap theme={null}
       global:
       daemonsetLogging:
         enabled: true
       customLogging:
         enabled: true
         scheme: https
         # host endpoint copied from elasticsearch console with https
         # and port number removed.
         host: "<host-URL>"
         port: "9243"
         # kubernetes secret containing credentials
         secretName: elasticcreds
       ```

    3. Add the following entry to your `values.yaml` file to disable internal logging:

       ```yaml wrap theme={null}
       tags:
          logging: false
       ```

    4. Run the following command to upgrade the Astro Private Cloud release version in the `values.yaml` file:

       ```bash wrap theme={null}
       helm upgrade -f values.yaml --version=2.0.0 --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
       ```
  </Tab>
</Tabs>

### View Airflow task logs in Elastic

1. On the Elastic dashboard in the **Elasticsearch Service** area, click the Deployment name.
2. Click **Menu** > **Discover**. The **Create index pattern** screen appears.
3. Enter `vector.*` if you use Vector sidecar logging. In the **Name** field, enter `@timestamp` in the **Timestamp field**, and then click **Create index pattern**.
4. Click **Menu** > **Dashboard** to view all of the Airflow task logs for your Deployment on Astronomer.