athal7/dotfiles / dot_agents/skills/elasticsearch
elasticsearch
VerifiedLoad when investigating production errors, latency, or trace data — triage order, index patterns, field names, and time-range syntax. Use before hand-rolling a query DSL call or guessing field names.
Load xh before querying. Use xh-es-search '<index-expression>' with JSON in the Bash PAYLOAD environment variable. xh-apm-error-groups builds its own payload.
The APM helper reads nested _source fields and pairs the first exception's type and message. A missing exception or trace in any sampled group fails the whole result; its diagnostic is safe to report but does not recover evidence or authorize dispatch.
The wrapper uses ES_URL when set; otherwise it derives https://<esHost> from chezmoi data. Configure the hostname with chezmoi and ensure the matching xh session is provisioned. It fails closed when esHost is missing or invalid.
Triage order
State the hypothesis, bound the time window, and identify the scope (one service / one endpoint / one user / system-wide) before querying anything. Write down what a "confirmed" answer looks like — otherwise absence of evidence reads as evidence of absence.
Don't start with logs. Work coarse to fine: error/rate spike (is it real?) → APM traces (which transaction) → APM errors (root-cause code path) → logs (raw context). trace.id links all three indices — pull it from an error or slow trace, then {"term": {"trace.id": "<id>"}} for the surrounding logs.
| Symptom | Look first |
|---|---|
| Slow page loads | Traces, sort transaction.duration.us desc |
| 500s spiking | Errors, group by error.grouping_key |
| One user affected | Logs, filter by user/session id |
| Periodic | Logs, time pattern in @timestamp |
| After a deploy | Errors, filter @timestamp after deploy |
Escalate rather than keep digging when: error rate stays above baseline >15min with no identified cause · a trace shows an external dependency timing out · errors reference a migration or schema change · you need production access you don't have.
Time range syntax
Pass time_range as a string like 15m, 1h, 24h, 7d. Translates to now-{value}{unit} in ES range filters.
Query logs
Search application logs. Index: logs-*. Sorted by @timestamp desc.
POST logs-*/_search
{"query":{"bool":{"must":[{"query_string":{"query":"YOUR LUCENE QUERY HERE"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","message","log.level","service.name","trace.id"],"sort":[{"@timestamp":"desc"}],"size":100}
Response fields: .hits.hits[]._source — extract @timestamp, log.level, service.name, message.
Add a service filter by inserting a term clause into the must array:
{"term": {"service.name": "my-service"}}
Query APM traces
Find slow transactions. Index: traces-apm*. Sorted by duration desc.
POST traces-apm*/_search
{"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":"now-1h"}}},{"range":{"transaction.duration.us":{"gte":500000}}}]}},"_source":["@timestamp","service.name","transaction.name","transaction.duration.us","transaction.result","trace.id"],"sort":[{"transaction.duration.us":"desc"}],"size":50}
Response fields: .hits.hits[]._source — extract @timestamp, service.name, transaction.name, transaction.duration.us (microseconds), transaction.result.
Query APM errors
Find exceptions and error groups. Index: logs-apm.error-*. Sorted by @timestamp desc.
POST logs-apm.error-*/_search
{"query":{"bool":{"must":[{"exists":{"field":"error.exception"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","error.exception.type","error.exception.message","error.grouping_key","service.name","transaction.name"],"sort":[{"@timestamp":"desc"}],"size":50}
Response fields: .hits.hits[]._source — extract @timestamp, error.exception.type, error.exception.message, service.name.
Tips
query_stringuses Lucene syntax:error AND timeout,level:ERROR,message:"connection refused"- To count by service: append
,"aggs":{"by_svc":{"terms":{"field":"service.name","size":10}}}to the query JSON and read.aggregations.by_svc.buckets trace.idlinks logs ↔ traces ↔ errors across indices
Kibana Dashboard API Gotchas
PUT /api/saved_objects/dashboard/:idreplaces ALL attributes. Read the full object first, modify onlypanelsJSON, and write everything back includingcontrolGroupInput,optionsJSON, etc. Omitting any attribute silently breaks panels.PUT /api/saved_objects/index-pattern/:idwipes thefieldsattribute if you only settitle/timeFieldName. To recreate safely, delete and usePOST /api/data_views/data_viewwhich auto-discovers fields.- Inline Lens panels referencing an
index-patternsaved object render blank if that object is corrupted. The resilient pattern isadHocDataViews+internalReferencesinsideembeddableConfig.attributes.state— self-contained, no external saved-object dependency. - ES transform
_updatecannot changepivot. Must stop, delete, recreate. If the dest index has historical data from rolled-over source indices, check_snapshotfirst.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |