notque/vexjoy-agent / skills/infrastructure/kubernetes
kubernetes
FlaggedKubernetes and infrastructure operations: cluster debugging, security and RBAC, manifests and Helm charts, Ansible, Prometheus/Grafana, Perses, OpenSearch/Elasticsearch, RabbitMQ, and cobaltcore tooling.
references/opensearch-elasticsearch/cluster-operations.md
View on GitHub---
description: Cluster health, shard allocation, capacity planning, rolling upgrades, and snapshot/restore operations
---
# OpenSearch/Elasticsearch Cluster Operations
> **Scope**: Cluster health management, shard allocation debugging, node roles, JVM heap tuning, rolling upgrades, and snapshot configuration. OpenSearch 2.x and Elasticsearch 8.x.
> **Version range**: OpenSearch 2.0+ / Elasticsearch 8.0+
> **Generated**: 2026-04-08
---
## Overview
Cluster operations have asymmetric consequences: misconfigured heap or shard counts are silent until load spikes. Yellow cluster status is tolerable; red is data loss risk. The most dangerous operations — DELETE index, update live mapping, shrink shards — are irreversible without snapshots. Every cluster configuration change requires a before/after snapshot.
---
## Pattern Table
| Pattern | Version | Use When | Prefer Another Pattern When |
|---------|---------|----------|------------|
| `cluster.routing.allocation.enable: all` | All versions | After maintenance window | Never set to `none` and forget |
| `indices.recovery.max_bytes_per_sec` | All versions | Limiting recovery bandwidth | Default is unlimited (saturates network) |
| Hot-warm-cold node roles | OS 2.0+ / ES 7.0+ | Mixed workload (active + archive data) | Single-tier small clusters |
| Cross-cluster replication | OS 1.1+ / ES 6.5+ | DR, geographic distribution | Simple single-cluster HA |
| Snapshot before destructive ops | Always | Before DELETE, reindex, mapping update | Never skip |
---
## Correct Patterns
### Diagnosing Yellow/Red Cluster Status
Start with allocation explain before guessing.
```bash
# Step 1: Overall health
GET /_cluster/health?pretty
# Step 2: Identify unassigned shards
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state:desc
# Step 3: Get authoritative explanation for unassigned shard
GET /_cluster/allocation/explain
{
"index": "your-index",
"shard": 0,
"primary": false
}
# Step 4: Check disk thresholds (common cause)
GET /_cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk*
# Common fix for disk threshold exceeded:
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "85%",
"cluster.routing.allocation.disk.watermark.high": "90%",
"cluster.routing.allocation.disk.watermark.flood_stage": "95%"
}
}
```
**Why**: `GET /_cluster/allocation/explain` tells you exactly why a shard won't assign (disk full, no eligible node, node excluded, etc.). Guessing without this leads to misdiagnosis.
---
### JVM Heap Configuration
```bash
# In jvm.options (or opensearch.yml for OS 2.12+):
# Set to 50% of available RAM, never exceed 31GB
-Xms16g
-Xmx16g
# Verify heap in use
GET /_nodes/stats?filter_path=nodes.*.jvm.mem
# Monitor GC pressure
GET /_nodes/stats?filter_path=nodes.*.jvm.gc.collectors.*.collection_time_in_millis
# GC time > 25% of wall clock = heap pressure
```
**Why**: JVM uses compressed ordinary object pointers (compressed OOPs) when heap is at or below ~31GB. Above this threshold, the JVM switches to 64-bit pointers — pointer size doubles, effective heap capacity drops by ~30%. Set heap to exactly 31g maximum; verify with `GET /_nodes` that compressed OOPs is active.
---
### Rolling Upgrade Procedure
```bash
# 1. Disable shard allocation before each node
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "primaries"
}
}
# 2. Flush syncronized (ES 7.6+ / OS: not needed — handled by upgrade)
POST /_flush
# 3. Stop node, upgrade, start node
systemctl stop opensearch
# ... upgrade package ...
systemctl start opensearch
# 4. Wait for node to rejoin
GET /_cat/nodes?v
# 5. Re-enable allocation
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": null
}
}
# 6. Wait for green before upgrading next node
GET /_cluster/health?wait_for_status=green&timeout=300s
# 7. Repeat for each node
```
**Why**: Disabling allocation before stopping a node prevents shard recovery storms. Without this, the cluster starts recovering replicas to other nodes as soon as the node goes down — only to cancel and re-recover when it comes back up. This wastes network and CPU.
---
### Snapshot Configuration
```bash
# Register S3 snapshot repository
PUT /_snapshot/backups
{
"type": "s3",
"settings": {
"bucket": "es-snapshots-prod",
"region": "us-east-1",
"base_path": "snapshots",
"compress": true,
"chunk_size": "1gb",
"server_side_encryption": true
}
}
# Create snapshot policy (automated)
PUT /_slm/policy/nightly-snapshots
{
"schedule": "0 30 1 * * ?",
"name": "<nightly-snap-{now/d}>",
"repository": "backups",
"config": {
"indices": ["*"],
"ignore_unavailable": false,
"include_global_state": true
},
"retention": {
"expire_after": "30d",
"min_count": 5,
"max_count": 50
}
}
# Verify latest snapshot
GET /_snapshot/backups/_all?pretty&s=start_time:desc
```
**Why**: `include_global_state: true` captures index templates, ILM policies, and cluster settings — not just data. Without global state, a full cluster restore requires manual recreation of all configuration.
---
## Pattern Catalog
### Keep JVM Heap at or Below 31GB
**Detection**:
```bash
# Check current heap for all nodes
GET /_nodes?filter_path=nodes.*.jvm.mem.heap_max_in_bytes
# Convert bytes to GB and flag if > 31GB
curl -s "$ES_HOST/_nodes/stats/jvm" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for nid, n in data['nodes'].items():
heap_gb = n['jvm']['mem']['heap_max_in_bytes'] / (1024**3)
flag = ' *** ABOVE 31GB ***' if heap_gb > 31 else ''
print(f\"{n['name']}: {heap_gb:.1f}GB{flag}\")
"
```
**Signal**:
```bash
# jvm.options
-Xms64g
-Xmx64g # 64GB — loses compressed OOPs
```
**Why this matters**: Above ~31GB, JVM object pointers are 64-bit instead of 32-bit compressed. Memory per object increases significantly. The JVM now needs a larger heap to hold the same amount of data. A 64GB heap may hold less effective data than a properly configured 31GB heap. GC pauses also increase with heap size.
**Preferred action**: Split heap across two nodes rather than increase past 31GB. For search-heavy workloads, two nodes with 15GB heap each generally outperform one node with 31GB because queries parallelize across shards on different nodes.
---
### Re-enable Shard Allocation After Maintenance
**Detection**:
```bash
# Check allocation settings — should be 'all' during normal operation
GET /_cluster/settings?filter_path=*.cluster.routing.allocation.enable
# Anything other than 'all' or null is suspect
```
**Signal**:
```bash
# Set during maintenance window, then forgotten
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "none"
}
}
# Maintenance completes, engineer leaves — setting persists
```
**Why this matters**: With `allocation.enable: none`, no new shard assignments occur. Adding a new node does nothing — no shards migrate to it. A data node failure causes replicas to become unassigned but not recovered. The cluster silently degrades toward red status.
**Preferred action**: After every maintenance operation, verify allocation is re-enabled:
```bash
PUT /_cluster/settings { "transient": { "cluster.routing.allocation.enable": null } }
GET /_cluster/health?wait_for_status=green&timeout=60s
```
---
### Set Replicas to 1+ on Production Indices
**Detection**:
```bash
# Find indices with number_of_replicas: 0
GET /_cat/indices?v&h=index,rep&s=rep:asc | head -20
# Any index with rep=0 in production is a risk
# Verify via settings
GET /_all/_settings?filter_path=*.settings.index.number_of_replicas
```
**Signal**:
```json
PUT /logs-2024-01
{
"settings": {
"number_of_replicas": 0
}
}
```
**Why this matters**: A node failure with 0 replicas causes data loss for all primary shards on that node. Cluster status goes RED. Documents indexed after the last snapshot are permanently lost.
**Preferred action**: `number_of_replicas: 1` minimum in production. Even for a single-node dev cluster, set to 0 explicitly with a comment explaining it's dev-only — don't let it be the default.
---
### Size Shard Count to Data Volume
**Detection**:
```bash
# Check shard sizes
GET /_cat/shards?v&h=index,shard,prirep,state,store&s=store:desc
# Calculate average shard size per index
GET /_cat/indices?v&h=index,pri,store.size
# Compute: store.size / pri = avg shard size
# Target: 20-50GB per shard
```
**Signal**:
```bash
# New index created with 5 shards for a 500MB dataset
PUT /tiny-index
{
"settings": { "number_of_shards": 5 }
}
# 5 shards of 100MB each — massive overhead for tiny data
```
**Why this matters**: Each shard has overhead: file descriptors, JVM memory in the shard metadata, cluster state size. 1000 shards of 1MB each consume as much overhead as 1000 shards of 50GB each — but the 1MB shards return results 100x slower due to cross-shard coordination cost on empty shards.
**Preferred action**: `number_of_shards: 1` for indices < 5GB. Use rollover for growing indices rather than pre-allocating many shards.
---
## Error-Fix Mappings
| Error Message | Root Cause | Fix |
|---------------|------------|-----|
| `cluster_block_exception: read-only index block` | Disk flood stage (95%) reached | Free disk space, then: `PUT /index/_settings {"index.blocks.read_only_allow_delete": null}` |
| `no_shard_available_action_exception` | All shards unassigned (cluster RED) | `GET /_cluster/allocation/explain` to diagnose; check disk space and node health |
| `primary shard is not active` | Primary shard lost (node down, no replica) | Restore from snapshot; use `GET /_cat/snapshots` |
| `circuit_breaking_exception: [parent]` | Coordinating node OOM during large operation | Reduce query scope; increase circuit breaker limit or heap |
| `TOO_MANY_REQUESTS/12/disk usage exceeded` | Disk watermark breach | Free disk > 15% or add nodes |
| Node not joining cluster | `cluster.name` mismatch or discovery misconfigured | Check `cluster.name` in `opensearch.yml` matches all nodes |
| `version conflict` on bulk index | Optimistic concurrency violation | Use `retry_on_conflict: 3` in bulk requests |
---
## Version-Specific Notes
| Version | Change | Impact |
|---------|--------|--------|
| OS 2.0 | Node role `data_content`, `data_hot`, `data_warm`, `data_cold` added | Explicit hot-warm-cold tier roles replace custom attributes |
| ES 8.0 | Security enabled by default | HTTP requires TLS; plain HTTP connections fail |
| OS 2.12 | `opensearch.yml` JVM heap options (replacing jvm.options) | Heap config can be in main config file |
| OS 2.4 | Segment replication GA | Replicas receive segment files instead of re-indexing — 30-50% less CPU |
---
## Detection Commands Reference
```bash
# Cluster health and unassigned shards
GET /_cluster/health?pretty
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state:desc
# JVM heap per node
GET /_nodes/stats?filter_path=nodes.*.jvm.mem.heap_max_in_bytes,nodes.*.name
# Allocation settings
GET /_cluster/settings?filter_path=*.cluster.routing.allocation.enable
# Indices with 0 replicas
GET /_cat/indices?v&h=index,rep | awk '$2 == "0" {print $1}'
# Shard size distribution
GET /_cat/shards?v&h=index,shard,prirep,store&s=store:desc | head -30
# Snapshot status
GET /_snapshot/_all
GET /_slm/policy
```
---
## See Also
- `index-management.md` — Mapping, ILM, and template configuration
- `query-optimization.md` — Query performance tuning
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |