Manage Elasticsearch log indices

Astro Private Cloud (APC) stores task logs and platform logs in Elasticsearch. APC 1.0 removed Kibana, so you administer the log indices with kubectl and the Elasticsearch REST API. This page shows you how to list indices, check their health, delete indices that you no longer need, and set the retention period that deletes old indices automatically.

The elasticsearch.<base-domain> endpoint doesn’t work for these tasks. The NGINX proxy in front of Elasticsearch allows only the _search, _count, _bulk, _cluster/health, and _cluster/state/version paths, and it rewrites the first three so that each caller reads only its own indices. The proxy denies every other path, including _cat and index deletion. Run the commands in this document from inside the cluster.

Prerequisites

kubectl access to the namespace where you installed the APC platform.
The name of your platform release. Run helm ls -A to find it.

Connect to the Elasticsearch API

Each Elasticsearch client Pod serves the REST API on port 9200. kubectl exec runs the request inside the Pod, so the network policy that protects Elasticsearch doesn’t block it.

Set the namespace and find a client Pod:

export NAMESPACE=<your-platform-namespace>
export ES_POD=$(kubectl get pods -n $NAMESPACE -l component=elasticsearch,role=client -o jsonpath='{.items[0].metadata.name}')

Confirm that the API answers:
kubectl exec -n $NAMESPACE $ES_POD -c es-client -- curl -s localhost:9200
The response includes the cluster name and the Elasticsearch version.

The rest of this document uses these two variables.

Index names

Vector creates one index for each Deployment for each day. The index name has three parts:

<prefix>.<deployment-release-name>.<YYYY.MM.DD>

For example, fluentd.spacious-orbit-4832.2026.02.14. The prefix depends on your logging configuration:

Configuration	Prefix
DaemonSet logging, which is the default	`fluentd`
Sidecar logging, when `global.logging.loggingSidecar.enabled` is `true`	`vector`
Any value set in `global.logging.indexNamePrefix`	That value

List indices and check their health

List every index with its health, document count, and size on disk:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s "localhost:9200/_cat/indices?v&h=health,status,index,docs.count,store.size&s=index"

The response looks like the following example:

health status index                                  docs.count store.size
green  open   fluentd.spacious-orbit-4832.2026.02.13     418293    412.8mb
green  open   fluentd.spacious-orbit-4832.2026.02.14     392014    388.1mb

An index is green when all of its shards are assigned, yellow when a replica shard is unassigned, and red when a primary shard is missing. A red index can’t serve all of its logs. To list only the indices that need attention, filter on health:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s "localhost:9200/_cat/indices?v&health=red"

To find the indices that use the most disk, sort by size:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s "localhost:9200/_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc"

To see the health of the cluster and how much disk each node uses:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- curl -s "localhost:9200/_cluster/health?pretty"
kubectl exec -n $NAMESPACE $ES_POD -c es-client -- curl -s "localhost:9200/_cat/allocation?v"

Delete an index

Deleting an index permanently deletes the logs in it. You can’t undo the operation. Forward your logs to external storage first if you must keep them. See Forward logs to Amazon S3.

Delete one index by name:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s -XDELETE "localhost:9200/fluentd.spacious-orbit-4832.2026.02.13"

Elasticsearch answers {"acknowledged":true}.

Delete several indices

Elasticsearch 8 sets action.destructive_requires_name to true, and APC keeps that default. A wildcard such as fluentd.* therefore fails with Wildcard expressions or all indices are not allowed. Name each index that you want to delete.

Review the names first. A GET request accepts a wildcard, so you can list the candidates:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s "localhost:9200/_cat/indices/fluentd.*.2026.01.*?h=index,store.size&s=index"

Delete the reviewed names in one request, separated by commas:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- \
  curl -s -XDELETE "localhost:9200/fluentd.spacious-orbit-4832.2026.01.02,fluentd.spacious-orbit-4832.2026.01.03"

Verify the result

After a delete, check the health of the cluster:

kubectl exec -n $NAMESPACE $ES_POD -c es-client -- curl -s "localhost:9200/_cluster/health?pretty"

status is green when every shard is assigned, and unassigned_shards is 0. List the indices again to confirm that the deleted ones are gone.

Delete old indices automatically

APC runs an Elasticsearch curator CronJob named <your-platform-release-name>-elasticsearch-curator. By default it runs at 01:00 every day and deletes indices that are more than 10 days old. Curator reads the date from the index name, so it only deletes indices that follow the <prefix>.<deployment-release-name>.<YYYY.MM.DD> pattern. To change the retention period, set the following values in your values.yaml file:

elasticsearch:
  curator:
    enabled: true
    schedule: "0 1 * * *"
    age:
      timestring: "%Y.%m.%d"
      unit: "days"
      unit_count: 10

Apply the change. See Apply a platform configuration change. To check the most recent run:

kubectl get jobs -n $NAMESPACE | grep elasticsearch-curator
kubectl logs -n $NAMESPACE -l app=elasticsearch-curator --tail=50

Overview

Architecture

Upgrade to Astro Private Cloud

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

Houston API

Platform Observability

Release notes

Version compatibility

Reference

Manage Elasticsearch log indices

Prerequisites

Connect to the Elasticsearch API

Index names

List indices and check their health

Delete an index

Delete several indices

Verify the result

Delete old indices automatically

​Prerequisites

​Connect to the Elasticsearch API

​Index names

​List indices and check their health

​Delete an index

​Delete several indices

​Verify the result

​Delete old indices automatically

Prerequisites

Connect to the Elasticsearch API

Index names

List indices and check their health

Delete an index

Delete several indices

Verify the result

Delete old indices automatically