> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Logs configuration

Astro Private Cloud (APC) provides centralized logging through Vector and Elasticsearch. Task logs, platform logs, and audit logs are collected by Vector and indexed in Elasticsearch for searchability and troubleshooting.

## Architecture

```mermaid actions={true} theme={null}
flowchart LR
    A[Airflow Components] --> V[Vector] --> E[Elasticsearch]
    E --> BYO([Bring your own visualization tool])
```

Vector runs as a DaemonSet collecting logs from all pods. Logs are shipped to Elasticsearch for storage and indexing. For log visualization, you can connect your own tools (Kibana, Grafana, OpenSearch Dashboards, etc.) to query Elasticsearch.

## Accessing logs

### Airflow UI

Task logs are accessible directly in the Airflow webserver UI:

1. Navigate to the Dag.
2. Click a task instance.
3. Click **Log**.

### Elasticsearch API

Query logs directly via Elasticsearch:

```bash wrap theme={null}
# Search for errors in the last hour
curl -X GET "https://elasticsearch.<base-domain>/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "log_level": "ERROR" } },
          { "range": { "@timestamp": { "gte": "now-1h" } } }
        ]
      }
    }
  }'
```

### BYO visualization

APC does not include a log visualization UI. Connect your preferred tool to Elasticsearch:

* **Kibana**: Deploy separately and point to the Elasticsearch endpoint.
* **Grafana**: Use the Elasticsearch data source.
* **OpenSearch Dashboards**: Use Elasticsearch API compatibility.

## Vector configuration

Vector is the log collection agent in APC 1.0.

### Enable Vector

```yaml wrap theme={null}
tags:
  logging: true

vectorEnabled: true
  vector:
    resources:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"
```

### Custom log parsing

Add custom transforms to parse Airflow log formats:

```yaml wrap theme={null}
vector:
  customConfig: |
    [transforms.parse_airflow]
    type = "remap"
    inputs = ["kubernetes_logs"]
    source = '''
    .parsed = parse_regex!(.message, r'^\[(?P<timestamp>.+)\] \{(?P<logger>.+)\} (?P<level>\w+) - (?P<message>.+)$')
    '''
```

### Logging sidecar

APC supports either DaemonSet or sidecar logging on a data plane cluster, but not both simultaneously. To use sidecar logging, you must first disable the Vector DaemonSet, then enable the sidecar:

```yaml wrap theme={null}
global:
  vectorEnabled: false
  loggingSidecar:
    enabled: true
    name: sidecar-log-consumer
    repository: quay.io/astronomer/ap-vector
    tag: 0.52.0
    resources:
      requests:
        cpu: "100m"
        memory: "386Mi"
```

## Elasticsearch configuration

### Enable Elasticsearch

```yaml expandable wrap theme={null}
tags:
  logging: true

elasticsearch:
  common:
    persistence:
      enabled: true

  client:
    replicas: 2
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

  data:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "100Gi"

  master:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
```

### Index lifecycle management

Configure log retention:

```yaml wrap theme={null}
elasticsearch:
  indexLifecycleManagement:
    enabled: true
    policies:
      - name: airflow-logs
        phases:
          hot:
            actions:
              rollover:
                max_size: 50gb
                max_age: 7d
          delete:
            min_age: 30d
            actions:
              delete: {}
```

## External logging

### Forward to external Elasticsearch

Send logs to your own Elasticsearch cluster:

```yaml wrap theme={null}
global:
  customLogging:
    enabled: true
    scheme: https
    host: "elasticsearch.example.com"
    port: "9200"
    secret: "es-credentials"
```

### Forward to S3

Archive logs to object storage:

```yaml wrap theme={null}
vector:
  sinks:
    s3:
      type: "aws_s3"
      inputs: ["kubernetes_logs"]
      bucket: "my-logs-bucket"
      region: "us-west-2"
      compression: "gzip"
      encoding:
        codec: "json"
```

### Forward to external systems

Configure Vector to send to any destination:

```yaml wrap theme={null}
vector:
  sinks:
    # Splunk
    splunk:
      type: "splunk_hec"
      inputs: ["kubernetes_logs"]
      endpoint: "https://splunk.example.com:8088"
      token: "${SPLUNK_TOKEN}"

    # Datadog
    datadog:
      type: "datadog_logs"
      inputs: ["kubernetes_logs"]
      default_api_key: "${DATADOG_API_KEY}"

    # Generic HTTP
    http:
      type: "http"
      inputs: ["kubernetes_logs"]
      uri: "https://logs.example.com/v1/logs"
      encoding:
        codec: "json"
```

## Deployment log settings

### Task log retention

Configure log groomer to manage disk usage:

```yaml wrap theme={null}
# In deployment values
scheduler:
  logGroomerSidecar:
    enabled: true
    retentionDays: 15
    frequencyMinutes: 15

workers:
  logGroomerSidecar:
    enabled: true
    retentionDays: 15
```

### Log level configuration

```yaml wrap theme={null}
env:
  - name: AIRFLOW__LOGGING__LOGGING_LEVEL
    value: "INFO"
  - name: AIRFLOW__LOGGING__FAB_LOGGING_LEVEL
    value: "WARNING"
```

## Querying logs

### Elasticsearch query examples

**Find task failures**:

```json wrap theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "match": { "log_level": "ERROR" } },
        { "match": { "kubernetes.labels.component": "worker" } }
      ]
    }
  }
}
```

**Search specific Dag**:

```json wrap theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "match": { "dag_id": "my_dag" } },
        { "match": { "task_id": "my_task" } }
      ]
    }
  }
}
```

**Filter by time range**:

```json wrap theme={null}
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2026-02-01T00:00:00",
        "lt": "2026-02-02T00:00:00"
      }
    }
  }
}
```

### Common log fields

| Field                         | Description                         |
| ----------------------------- | ----------------------------------- |
| `kubernetes.namespace_name`   | Deployment namespace                |
| `kubernetes.labels.component` | Component (scheduler, worker, etc.) |
| `kubernetes.pod_name`         | Pod name                            |
| `dag_id`                      | DAG identifier                      |
| `task_id`                     | Task identifier                     |
| `log_level`                   | DEBUG, INFO, WARNING, ERROR         |
| `@timestamp`                  | Log timestamp                       |

## Troubleshooting

### Logs aren't appearing

1. **Check Vector is running**:
   ```bash wrap theme={null}
   kubectl get pods -n astronomer -l app=vector
   ```

2. **Check Elasticsearch health**:
   ```bash wrap theme={null}
   kubectl exec -n astronomer elasticsearch-0 -- \
     curl -s localhost:9200/_cluster/health
   ```

3. **Verify Vector logs**:
   ```bash wrap theme={null}
   kubectl logs -n astronomer -l app=vector --tail=100
   ```

### High disk usage

1. Enable index lifecycle management.
2. Reduce retention period.
3. Increase Elasticsearch storage.
4. Forward logs to external storage (S3).

### Slow queries

1. Add index patterns for common searches.
2. Increase Elasticsearch resources.
3. Reduce log verbosity.

## Security

### Access control

Elasticsearch access is restricted to platform components. For external access, configure authentication:

```yaml wrap theme={null}
elasticsearch:
  auth:
    enabled: true
    secretName: "es-credentials"
```

### Log redaction

Redact sensitive data before indexing:

```yaml wrap theme={null}
vector:
  transforms:
    redact:
      type: "remap"
      source: '''
      .message = replace(.message, r'password=\S+', "password=***")
      .message = replace(.message, r'api_key=\S+', "api_key=***")
      '''
```

## Best practices

* Set appropriate retention based on compliance requirements.
* Use log levels wisely - avoid DEBUG in production.
* Enable log groomer to prevent disk exhaustion on Airflow pods.
* Forward logs externally for long-term retention and compliance.
* Monitor Elasticsearch health and disk usage.
* Use your preferred visualization tool - deploy Kibana, Grafana, or other tools separately.