This article focuses on automated cleanup for Elasticsearch log indexes. The core approach is to use time-based indexes and delete expired data with ILM, crontab, or Curator. This helps prevent disk exhaustion, reduce the risk of manual deletion, and improve operational efficiency. Keywords: Elasticsearch, ILM, index lifecycle.
The technical specification snapshot provides a quick overview
| Parameter | Description |
|---|---|
| Core system | Elasticsearch |
| Primary languages | JSON DSL, Bash, YAML |
| Management protocol | HTTP REST API |
| Typical scenarios | Cleanup for logs, monitoring, and behavioral data |
| Official capability | ILM (Index Lifecycle Management) |
| Third-party tool | Curator |
| Star count | Original data not provided |
| Core dependencies | Elasticsearch, curl, crontab, elasticsearch-curator |
Elasticsearch automated cleanup should be built on time-based index design
In production environments, logs, monitoring data, and event-tracking data are continuously written into Elasticsearch. If you keep all of that data in a single index for too long, disk utilization, shard size, and query performance will quickly degrade.
Elasticsearch does not recommend frequent deletion of individual documents, because that triggers segment merges and additional I/O. A more efficient approach is to create indexes by day or by another fixed interval, then delete the entire index after it expires.
The core cleanup workflow can be abstracted into a fixed closed loop
- Write data into daily indexes.
- Let the lifecycle policy monitor index age.
- Automatically delete indexes after the retention period expires.
- Reclaim disk space and keep search performance stable.
PUT _ilm/policy/logs-7d-delete-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "1d" // Roll over the index once per day
}
}
},
"delete": {
"min_age": "7d", // Enter the delete phase after 7 days
"actions": {
"delete": {} // Delete the entire old index directly
}
}
}
}
}This policy defines a standard log lifecycle with daily rollover and a 7-day retention period.
ILM is the official first choice and the best fit for production environments
ILM is a built-in Elasticsearch capability. It does not depend on an external scheduler, and you do not need to maintain deletion scripts yourself. It works well for most time-series data scenarios, especially logs and monitoring data.
Compared with manual deletion, ILM provides policy-driven management, auditability, and a lower risk of accidental deletion. It also works seamlessly with rollover, warm, cold, and delete phases.
The index template must bind both the lifecycle policy and the write alias
A policy alone is not enough. The template must bind the lifecycle policy and the rollover alias to the target index pattern. Otherwise, newly created indexes will not automatically inherit the rules.
PUT _index_template/template-logs
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"index.lifecycle.name": "logs-7d-delete-policy", // Bind the ILM policy
"index.lifecycle.rollover_alias": "logs-alias" // Specify the rollover write alias
}
}
}This step determines whether new indexes can automatically inherit the lifecycle rules.
Initializing the write index is the required starting point for ILM
Before using rollover, you must first create an initial index and point the write alias to it. After that, applications only need to write to the alias instead of writing directly to a specific index name.
PUT logs-000001
{
"aliases": {
"logs-alias": {
"is_write_index": true // Set the current index as the write target for the alias
}
}
}After this configuration is in place, ILM can take over index rollover and expired-index deletion.
Scheduled deletion scripts suit lightweight environments but are less reliable than ILM
If ILM is not enabled in the current cluster, or if indexes already follow a strict date-based naming convention such as logs-2025.12.01, you can implement simplified cleanup with a script and crontab.
This approach is quick to implement and easy to learn, but it depends on an external scheduler and carries a higher risk of accidental deletion than ILM. It is better suited for test environments, small clusters, or temporary transitional solutions.
A Bash script can delete the target index precisely by date
#!/bin/bash
# Calculate the date from 7 days ago to build the expired index name
expired_date=$(date -d '7 days ago' +%Y.%m.%d)
# Delete the log index for the corresponding date
curl -X DELETE "http://localhost:9200/logs-${expired_date}" \
-u elastic:123456This script deletes one expired log index per day by matching the date exactly.
# Run the cleanup script at 1:00 AM every day to avoid peak business hours
0 1 * * * /root/clean-es.shThis crontab rule schedules index deletion during off-peak hours.
Curator is better suited for enterprise scenarios that require complex filtering
When cleanup rules depend on more than just date, and also involve prefixes, states, shards, or multi-cluster batch governance, Curator is more maintainable than plain Bash. In essence, it wraps Elasticsearch operational actions into policy-based definitions.
However, Curator introduces additional deployment and configuration overhead. If ILM already meets your requirements, you should still prioritize the native Elasticsearch solution.
Curator configuration is well suited for auditable batch deletion logic
actions:
1:
action: delete_indices
filters:
- filtertype: pattern
value: logs-* # Process only log indexes
- filtertype: age
source: name
direction: older
unit: days
unit_count: 7 # Delete indexes older than 7 daysThis YAML configuration expresses deletion criteria declaratively, which makes it a good fit for standardized operations workflows.
# Run the Curator configuration to clean up expired indexes in bulk according to the rules
curator config.yml actions.ymlThis command applies the defined cleanup policy to the Elasticsearch cluster.
The trade-offs between the three cleanup approaches should be based on stability and maintenance cost
| Approach | Implementation complexity | Stability | Best-fit scenario | Recommendation |
|---|---|---|---|---|
| ILM lifecycle management | Low | Very high | Standard log governance in production | ⭐⭐⭐⭐⭐ |
| crontab + script | Low | Medium | Small-scale or temporary solutions | ⭐⭐⭐ |
| Curator | Medium | High | Complex batch governance scenarios | ⭐⭐⭐⭐ |
The core decision is simple: if ILM can solve the problem, do not start with scripts; if you need more complex filtering, then consider Curator.
Production best practices must focus on performance, safety, and recoverability
Log data should always be split into indexes by day or by rollover rules. This makes deletion an index-level operation, which minimizes cost and side effects.
Before deletion, you should keep snapshots when necessary, especially for audit logs, transaction logs, and compliance retention data. Data lifecycle management is not just about deleting data; it is about retiring data in a traceable way.
The following checklist is practical for production use
- Roll log indexes daily or through data streams.
- Enable ILM first instead of relying on manual scripts.
- Set log retention to 7 to 15 days in most cases.
- Schedule deletion during low-traffic hours after midnight.
- Keep required snapshots before cleanup.
- Avoid running
delete by querydirectly on massive historical datasets.
A flowchart helps explain the real ILM execution path
AI Visual Insight: This diagram presents the ILM implementation path as a linear workflow. You first create a lifecycle policy, then bind it to an index template. Incoming writes flow into the current write index. When the index reaches the rollover condition, Elasticsearch creates a new index. After the old index exceeds the retention period, it enters the delete phase and is removed automatically, resulting in reclaimed disk space and lower cluster load.
The FAQ provides structured answers to common operational questions
FAQ 1: Why is directly deleting Elasticsearch documents not recommended?
Direct document deletion triggers segment-level delete markers and subsequent merges, which creates high I/O cost and noticeable performance fluctuations. Deleting an entire index by time range is faster, more stable, and better aligned with time-series data management.
FAQ 2: ILM is configured, but indexes are not deleted automatically. What should you check first?
First, verify that the index is actually bound to index.lifecycle.name. Then confirm that rollover_alias is set correctly and that the initial write index has been created. Finally, check whether the index age has already reached the min_age threshold.
FAQ 3: How should you choose between ILM, Curator, and script-based deletion?
Use ILM first in production. Choose Curator when you need cross-index operations, complex filtering, and centralized orchestration. Use script-based deletion only in lightweight environments or as a temporary transition. The principle is straightforward: native first, extensions second, scripts last.
Core summary: This article systematically reconstructs the Elasticsearch automated data cleanup strategy. It explains the practical boundaries, implementation steps, and production best practices for ILM lifecycle management, scheduled deletion scripts, and Curator, helping teams avoid disk exhaustion, uncontrolled index growth, and the risks of manual deletion.