Skip to content

athal7/dotfiles / dot_agents/skills/elasticsearch

elasticsearch

Verified

Load when investigating production errors, latency, or trace data — triage order, index patterns, field names, and time-range syntax. Use before hand-rolling a query DSL call or guessing field names.

Load xh before querying. Use xh-es-search '<index-expression>' with JSON in the Bash PAYLOAD environment variable. xh-apm-error-groups builds its own payload. The APM helper reads nested _source fields and pairs the first exception's type and message. A missing exception or trace in any sampled group fails the whole result; its diagnostic is safe to report but does not recover evidence or authorize dispatch.

The wrapper uses ES_URL when set; otherwise it derives https://<esHost> from chezmoi data. Configure the hostname with chezmoi and ensure the matching xh session is provisioned. It fails closed when esHost is missing or invalid.

Triage order

State the hypothesis, bound the time window, and identify the scope (one service / one endpoint / one user / system-wide) before querying anything. Write down what a "confirmed" answer looks like — otherwise absence of evidence reads as evidence of absence.

Don't start with logs. Work coarse to fine: error/rate spike (is it real?) → APM traces (which transaction) → APM errors (root-cause code path) → logs (raw context). trace.id links all three indices — pull it from an error or slow trace, then {"term": {"trace.id": "<id>"}} for the surrounding logs.

SymptomLook first
Slow page loadsTraces, sort transaction.duration.us desc
500s spikingErrors, group by error.grouping_key
One user affectedLogs, filter by user/session id
PeriodicLogs, time pattern in @timestamp
After a deployErrors, filter @timestamp after deploy

Escalate rather than keep digging when: error rate stays above baseline >15min with no identified cause · a trace shows an external dependency timing out · errors reference a migration or schema change · you need production access you don't have.

Time range syntax

Pass time_range as a string like 15m, 1h, 24h, 7d. Translates to now-{value}{unit} in ES range filters.

Query logs

Search application logs. Index: logs-*. Sorted by @timestamp desc.

POST logs-*/_search
{"query":{"bool":{"must":[{"query_string":{"query":"YOUR LUCENE QUERY HERE"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","message","log.level","service.name","trace.id"],"sort":[{"@timestamp":"desc"}],"size":100}

Response fields: .hits.hits[]._source — extract @timestamp, log.level, service.name, message.

Add a service filter by inserting a term clause into the must array:

{"term": {"service.name": "my-service"}}

Query APM traces

Find slow transactions. Index: traces-apm*. Sorted by duration desc.

POST traces-apm*/_search
{"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":"now-1h"}}},{"range":{"transaction.duration.us":{"gte":500000}}}]}},"_source":["@timestamp","service.name","transaction.name","transaction.duration.us","transaction.result","trace.id"],"sort":[{"transaction.duration.us":"desc"}],"size":50}

Response fields: .hits.hits[]._source — extract @timestamp, service.name, transaction.name, transaction.duration.us (microseconds), transaction.result.

Query APM errors

Find exceptions and error groups. Index: logs-apm.error-*. Sorted by @timestamp desc.

POST logs-apm.error-*/_search
{"query":{"bool":{"must":[{"exists":{"field":"error.exception"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}},"_source":["@timestamp","error.exception.type","error.exception.message","error.grouping_key","service.name","transaction.name"],"sort":[{"@timestamp":"desc"}],"size":50}

Response fields: .hits.hits[]._source — extract @timestamp, error.exception.type, error.exception.message, service.name.

Tips

  • query_string uses Lucene syntax: error AND timeout, level:ERROR, message:"connection refused"
  • To count by service: append ,"aggs":{"by_svc":{"terms":{"field":"service.name","size":10}}} to the query JSON and read .aggregations.by_svc.buckets
  • trace.id links logs ↔ traces ↔ errors across indices

Kibana Dashboard API Gotchas

  • PUT /api/saved_objects/dashboard/:id replaces ALL attributes. Read the full object first, modify only panelsJSON, and write everything back including controlGroupInput, optionsJSON, etc. Omitting any attribute silently breaks panels.
  • PUT /api/saved_objects/index-pattern/:id wipes the fields attribute if you only set title/timeFieldName. To recreate safely, delete and use POST /api/data_views/data_view which auto-discovers fields.
  • Inline Lens panels referencing an index-pattern saved object render blank if that object is corrupted. The resilient pattern is adHocDataViews + internalReferences inside embeddableConfig.attributes.state — self-contained, no external saved-object dependency.
  • ES transform _update cannot change pivot. Must stop, delete, recreate. If the dest index has historical data from rolled-over source indices, check _snapshot first.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall