Logs configuration - Astronomer

Astro Private Cloud (APC) provides centralized logging through Vector and Elasticsearch. Vector collects Airflow task logs and Airflow component logs, and indexes them in Elasticsearch for search and troubleshooting.

This page covers the Vector DaemonSet pipeline that ships task and component logs. The APC API control plane audit events have a separate Vector sidecar with its own configuration and supported sinks. See the APC API audit logging overview.

Architecture

Vector runs as a DaemonSet on every node. It collects logs from Pods that carry an Airflow component label, such as scheduler, worker, triggerer, api-server, webserver, dag-processor, dag-server, and git-sync-relay. It doesn’t collect logs from platform components such as the APC API or the Registry. Vector ships the logs to Elasticsearch for storage and indexing. For log visualization, you can connect your own tools (Kibana, Grafana, OpenSearch Dashboards) to query Elasticsearch.

Access logs

Airflow UI

Task logs are accessible directly in the Airflow UI:

Go to the Dag run.
Click a task instance.
Click Log.

Elasticsearch API

Vector writes one index per Deployment per day. The index name is <index-name-prefix>.<deployment-release-name>.<date>. The prefix defaults to fluentd when the DaemonSet ships the logs, and to vector when the logging sidecar ships them. Set global.logging.indexNamePrefix to use a different prefix. Query logs directly through Elasticsearch:

# Search one Deployment's indices for errors in the last hour
curl -X GET "https://elasticsearch.<base-domain>/fluentd.<deployment-release-name>.*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "range": { "level_numeric": { "gte": 40 } } },
          { "range": { "@timestamp": { "gte": "now-1h" } } }
        ]
      }
    }
  }'

On a data plane that uses a domain prefix, the host is elasticsearch.<domain-prefix>.<base-domain>.

BYO visualization

APC doesn’t include a log visualization UI. Connect your preferred tool to Elasticsearch:

Kibana: Deploy separately and point to the Elasticsearch endpoint.
Grafana: Use the Elasticsearch data source.
OpenSearch Dashboards: Use Elasticsearch API compatibility.

Vector configuration

Vector is the log collection agent in APC.

Enable Vector

tags:
  logging: true

global:
  daemonsetLogging:
    enabled: true

vector:
  vector:
    resources:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"

The vector key appears twice. The outer key is the subchart, and the inner key is the container in the DaemonSet Pod.

Logging sidecar

APC supports either DaemonSet or sidecar logging on a data plane cluster, but not both at the same time. To use sidecar logging, first disable the Vector DaemonSet, then enable the sidecar:

global:
  daemonsetLogging:
    enabled: false
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
      resources:
        requests:
          cpu: "100m"
          memory: "384Mi"
        limits:
          cpu: "100m"
          memory: "384Mi"

The chart sets repository and tag for the sidecar image. Override them only when you use a private registry or a specific Vector version. For the complete procedure, see Export logs using container sidecars.

Custom log parsing

The chart generates the Vector DaemonSet configuration and mounts it as a read-only ConfigMap. You can’t change its sources or transforms with a Helm value. The one supported addition is an extra sink. See Forward to external systems. To change how logs are parsed, use the logging sidecar with your own Vector configuration:

Set global.logging.loggingSidecar.customConfig to true.
Write a complete Vector configuration file.
Create the sidecar-config secret from that file in the platform namespace, annotate the secret with astronomer.io/commander-sync, and run the config syncer job to copy it into the Deployment namespaces.

customConfig: true doesn’t create a configuration for you. It stops APC from creating the default sidecar configuration, so the sidecar has no configuration until you create and sync the secret. Do all three steps together.

For the full procedure, with example configurations for Elasticsearch, Datadog, and Honeycomb, see Customize Vector logging sidecars.

Elasticsearch configuration

Enable Elasticsearch

tags:
  logging: true

elasticsearch:
  common:
    persistence:
      enabled: true

  client:
    replicas: 2
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

  data:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "100Gi"

  master:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "20Gi"

Log retention

An Elasticsearch curator CronJob deletes log indices that are more than 10 days old. To change the retention period:

elasticsearch:
  curator:
    enabled: true
    # When the CronJob runs
    schedule: "0 1 * * *"
    age:
      timestring: "%Y.%m.%d"
      # Delete indices older than 10 days
      unit: "days"
      unit_count: 10

To inspect or delete indices yourself, see Manage Elasticsearch log indices.

External logging

Forward to external Elasticsearch

Send logs to your own Elasticsearch cluster:

global:
  customLogging:
    enabled: true
    scheme: https
    host: "elasticsearch.example.com"
    port: "9200"
    secret: "es-credentials"

For the full procedure, including how to store the credentials in a Kubernetes secret, see Use an external Elasticsearch instance for Airflow task log management.

Forward to S3

Archive logs to object storage with an extra sink. The inputs value must name a transform from the generated DaemonSet configuration. The last transform in that pipeline is transform_remove_fields:

vector:
  extraSinks:
    s3_logs:
      type: aws_s3
      inputs:
        - transform_remove_fields
      bucket: "my-logs-bucket"
      region: "us-west-2"
      compression: gzip
      encoding:
        codec: json

The sink also needs AWS credentials. For the complete procedure, see Forward logs to Amazon S3.

Forward to external systems

Use vector.extraSinks to send logs to any destination that Vector supports. APC adds these sinks to the generated configuration and keeps the Elasticsearch sink:

vector:
  extraSinks:
    # Splunk
    splunk:
      type: splunk_hec
      inputs:
        - transform_remove_fields
      endpoint: "https://splunk.example.com:8088"
      token: "${SPLUNK_TOKEN}"

    # Datadog
    datadog:
      type: datadog_logs
      inputs:
        - transform_remove_fields
      default_api_key: "${DATADOG_API_KEY}"

    # Generic HTTP
    http:
      type: http
      inputs:
        - transform_remove_fields
      uri: "https://logs.example.com/v1/logs"
      encoding:
        codec: json

Vector reads ${SPLUNK_TOKEN} and ${DATADOG_API_KEY} from its own environment. Add environment variables as key-value pairs:

vector:
  extraEnv:
    SPLUNK_TOKEN: <your-splunk-token>

vector.extraEnv takes literal values only. It can’t read a value from a Kubernetes secret.

Deployment log settings

Task log retention

The log groomer sidecar deletes old task log files from the Airflow Pods. deployments.helm.airflow.*.logGroomerSidecar is a deployments.* setting, so cluster configuration is the final layer here: the cluster value wins when both values.yaml and a cluster override are set. See Configure Astro Private Cloud for the precedence rules. To change retention for one data plane cluster, add the following to that cluster’s Configuration Override in the Astro UI, or pass it as deploymentsConfigOverride on the updateCluster Houston API mutation. See Update data plane cluster configurations.

{
  "helm": {
    "airflow": {
      "scheduler": {
        "logGroomerSidecar": {
          "enabled": true,
          "retentionDays": 15,
          "frequencyMinutes": 15
        }
      },
      "workers": {
        "logGroomerSidecar": {
          "enabled": true,
          "retentionDays": 15,
          "frequencyMinutes": 15
        }
      }
    }
  }
}

To change the platform default for clusters that don’t have their own saved value for this key, update values.yaml and run a Helm upgrade instead:

astronomer:
  houston:
    config:
      deployments:
        helm:
          airflow:
            scheduler:
              logGroomerSidecar:
                enabled: true
                retentionDays: 15
                frequencyMinutes: 15
            workers:
              logGroomerSidecar:
                enabled: true
                retentionDays: 15
                frequencyMinutes: 15

Log level configuration

Set the Airflow log level with environment variables on a Deployment. See Environment variables.

Key	Example value
`AIRFLOW__LOGGING__LOGGING_LEVEL`	`INFO`
`AIRFLOW__LOGGING__FAB_LOGGING_LEVEL`	`WARNING`

Reduce excessive Dag processor logging

In multi-deployment environments (for example, 1,000 or more Deployments each running multiple Dags), Deployments with Dags that have import errors can produce continuous, repetitive scheduler and Dag processor log entries every few seconds. The setting that reduces this noise depends on the Airflow version your Deployment runs.

Airflow 3

Airflow 3 uses [dag_processor] bundle_refresh_check_interval to control how often the Dag processor checks whether any Dag bundle is ready for a refresh, either because the bundle’s own refresh_interval has elapsed or because another Dag processor has seen a newer version of the bundle. A lower value keeps multiple Dag processors in sync with each other faster, but generates more frequent log activity. The default is 5 seconds. Increase this value to reduce how often the check and its associated logging occur. Set the following environment variable on the affected Deployment:

AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL=60

Airflow 2

Airflow 2 uses [logging] dag_processor_manager_log_stdout to control whether Dag processor manager logs are also sent to stdout, in addition to the dedicated manager log file (dag_processor_manager.log under the configured log directory). The default is False. In containerized Deployments where log aggregation or shipping only captures container stdout (for example, Vector or Fluentd), keeping this set to False prevents the noisy, frequently repeating manager output from being shipped:

AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_STDOUT=False

Query logs

Elasticsearch query examples

Find task failures

level_numeric is 40 for errors and 50 for critical events:

{
  "query": {
    "bool": {
      "must": [
        { "range": { "level_numeric": { "gte": 40 } } },
        { "match": { "component": "worker" } }
      ]
    }
  }
}

Search specific Dag

{
  "query": {
    "bool": {
      "must": [
        { "match": { "dag_id": "my_dag" } },
        { "match": { "task_id": "my_task" } }
      ]
    }
  }
}

Filter by time range

{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2026-02-01T00:00:00",
        "lt": "2026-02-02T00:00:00"
      }
    }
  }
}

Common log fields

Field	Description
`component`	Airflow component, such as `scheduler` or `worker`
`release`	Deployment release name
`namespace`	Deployment namespace
`workspace`	Workspace that owns the Deployment
`dag_id`	Dag identifier
`task_id`	Task identifier
`log_id`	Composite identifier of the Dag, task, run, and attempt
`log_type`	`task`, `system`, `component`, or `dag_parse`. See Log types.
`level`	Log level as the component wrote it
`level_numeric`	Log level as a number: 10 debug, 20 info, 30 warning, 40 error, 50 critical
`event`	Log message
`@timestamp`	Log timestamp

Vector removes the kubernetes, host, and file fields before it sends a log to Elasticsearch, so you can’t search on them. Use component, namespace, and release instead.

Log types

The log_type value a log gets depends on which Vector pipeline collected it:

`log_type`	Where it comes from
`task`	A task log, from either pipeline. Set whenever the log has a `dag_id`, or when the Airflow 3 log file path contains one.
`system`	A non-task log from a container’s stdout. This is what scheduler, API server, worker, and triggerer logs get on Airflow 2 Deployments.
`component`	A non-task log from an Airflow 3 Deployment’s log files.
`dag_parse`	A Dag processing log from an Airflow 3 Deployment’s log files.

On Airflow 2 Deployments, component logs are log_type: system, not log_type: component. To find logs for a specific Airflow component on any Airflow version, filter on the component field instead of log_type.

Troubleshooting

Logs aren’t appearing

Check Vector is running:

kubectl get pods -n astronomer -l app=vector

Check Elasticsearch health:

ES_POD=$(kubectl get pods -n astronomer -l component=elasticsearch,role=client -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n astronomer $ES_POD -c es-client -- \
  curl -s localhost:9200/_cluster/health

Verify Vector logs:

kubectl logs -n astronomer -l app=vector --tail=100

High disk usage

Reduce elasticsearch.curator.age.unit_count to delete indices sooner.
Delete the largest indices. See Manage Elasticsearch log indices.
Increase elasticsearch.data.persistence.size.
Reduce log verbosity.
Forward logs to external storage (S3) for long-term retention.

Slow queries

Query one Deployment’s indices instead of all indices.
Narrow the time range.
Increase Elasticsearch resources.

Security

Access control

The chart has no Helm value that turns on Elasticsearch authentication. Airflow components read task logs through the nginx-es proxy in the platform namespace. The proxy authenticates each request with the APC API and rewrites the query so that a Deployment reads only its own indices. It allows the _search, _count, _bulk, _cluster/health, and _cluster/state/version paths, and denies every other path. The elasticsearch.<base-domain> ingress goes to Elasticsearch directly. Restrict access to that host with your ingress controller and your network policies. See Network configuration.

Log redaction

You can’t add a transform to the Vector DaemonSet. To redact values before Elasticsearch indexes them, use the logging sidecar with a custom configuration and add a remap transform to it:

transforms:
  redact:
    type: remap
    inputs:
      - transform_airflow_logs
    source: |
      if is_string(.message) {
        .message = replace(string!(.message), r'password=\S+', "password=***")
        .message = replace(string!(.message), r'api_key=\S+', "api_key=***")
      }

The transform is part of the complete sidecar configuration file, so the next transform or the sink must name redact in its inputs. See Custom log parsing.

Best practices

Set the curator retention period from your compliance requirements.
Use log levels wisely — avoid DEBUG in production.
Enable the log groomer to prevent disk exhaustion on Airflow Pods.
Forward logs externally for long-term retention and compliance.
Monitor Elasticsearch health and disk usage.
Use your preferred visualization tool — deploy Kibana, Grafana, or other tools separately.

​Architecture

​Access logs

​Airflow UI

​Elasticsearch API

​BYO visualization

​Vector configuration

​Enable Vector

​Logging sidecar

​Custom log parsing

​Elasticsearch configuration

​Enable Elasticsearch

​Log retention

​External logging

​Forward to external Elasticsearch

​Forward to S3

​Forward to external systems

​Deployment log settings

​Task log retention

​Log level configuration

​Reduce excessive Dag processor logging

​Airflow 3

​Airflow 2

​Query logs

​Elasticsearch query examples

​Find task failures

​Search specific Dag

​Filter by time range

​Common log fields

​Log types

​Troubleshooting

​Logs aren’t appearing

​High disk usage

​Slow queries

​Security

​Access control

​Log redaction

​Best practices

Architecture

Access logs

Airflow UI

Elasticsearch API

BYO visualization

Vector configuration

Enable Vector

Logging sidecar

Custom log parsing

Elasticsearch configuration

Enable Elasticsearch

Log retention

External logging

Forward to external Elasticsearch

Forward to S3

Forward to external systems

Deployment log settings

Task log retention

Log level configuration

Reduce excessive Dag processor logging

Airflow 3

Airflow 2

Query logs

Elasticsearch query examples

Find task failures

Search specific Dag

Filter by time range

Common log fields

Log types

Troubleshooting

Logs aren’t appearing

High disk usage

Slow queries

Security

Access control

Log redaction

Best practices