ericrisco/rsc-harness / skills/observability
observability
VerifiedUse when instrumenting a service from the inside so an incident can be explained from telemetry alone — wiring OpenTelemetry logs, metrics and traces, standing up a Collector, exporting via OTLP, and defining telemetry-driven alerts. NOT outside-in uptime probes, on-call rotation, or who-gets-paged (that is `monitoring`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.0 KB |
| evals/README.md | markdown | 1.3 KB |
| evals/cases.yaml | config | 3.8 KB |
| references/collector-config.md | markdown | 4.3 KB |
| references/instrumentation-recipes.md | markdown | 4.8 KB |
| scripts/verify.shexecutable | script | 6.2 KB |
references/collector-config.md
View on GitHub# Collector config — fuller recipes
Branch-specific Collector depth offloaded from `SKILL.md`. Shapes target Collector **v0.153.0**. An exporter or processor is inert until it is named in a `service.pipelines` entry.
## Gateway vs agent topology
Two deployment shapes; most production setups run both as a tiered pipeline.
- **Agent** — one Collector per host/pod (sidecar or DaemonSet), close to the app. Cheap network hop, does fast local work: `memory_limiter`, `batch`, `resourcedetection`. Forwards via OTLP to the gateway.
- **Gateway** — a horizontally-scaled standalone tier. Does the expensive, stateful work that needs a *whole-trace* view: `tail_sampling`, cross-service `redaction`, per-tenant routing, backend fan-out. Scale it independently of your apps.
Rule: do per-span/per-host work in the agent, do whole-trace and policy work in the gateway. Tail sampling in an agent is wrong — an agent never sees a full trace.
```yaml
# agent (DaemonSet) → forwards everything to the gateway
exporters:
otlp/gateway: { endpoint: otel-gateway:4317, tls: { insecure: false } }
service:
pipelines:
traces: { receivers: [otlp], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp/gateway] }
metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/gateway] }
logs: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/gateway] }
```
## Tail sampling (gateway only)
Head sampling decides at span start, before you know if the request was interesting. **Tail sampling** buffers the whole trace and decides at the end — so you keep every error and every slow trace, and sample the boring fast ones. It must run on the gateway, after a `groupbytrace`-style stage so all spans of a trace land on the same Collector.
```yaml
processors:
tail_sampling:
decision_wait: 10s # how long to buffer a trace before deciding
num_traces: 100000
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] } # never drop a failed trace
- name: keep-slow
type: latency
latency: { threshold_ms: 1000 } # never drop a slow trace
- name: sample-the-rest
type: probabilistic
probabilistic: { sampling_percentage: 5 } # 5% of the boring ones
```
## Redaction / attributes for PII
Strip sensitive data before it leaves your network. `redaction` masks values by regex; `attributes` deletes or hashes specific keys.
```yaml
processors:
redaction:
allow_all_keys: true
blocked_values:
- "\\b[0-9]{13,16}\\b" # card PANs
- "\\b[\\w.+-]+@[\\w-]+\\.[\\w.-]+\\b" # emails
summary: debug # report what it masked, at debug
attributes/pii:
actions:
- { key: user.email, action: delete }
- { key: user.id, action: hash } # keep joinable, not readable
```
Wire both into the pipeline's `processors` list before the exporter, or they do nothing.
## Multi-backend fan-out (LGTM + a vendor in parallel)
A pipeline's `exporters` list is a fan-out — list two and every item goes to both. Useful for migrating off a vendor, or keeping a cheap self-hosted copy beside a managed one.
```yaml
exporters:
otlphttp/tempo: { endpoint: http://tempo:4318 }
otlphttp/honeycomb:
endpoint: https://api.honeycomb.io
headers: { "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" }
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo, otlphttp/honeycomb] # both, in parallel
```
LGTM split for reference: **Loki** logs, **Tempo** traces, **Mimir/Prometheus** metrics, **Grafana** to view.
## resourcedetection + env-var overrides
`resourcedetection` auto-fills `host.*`, `cloud.*`, `k8s.*`, `container.*` resource attributes so you don't hand-set them per deploy. Combine with env-var interpolation (`${env:VAR}`) so the same config file works across environments — never bake secrets or per-env endpoints into the file.
```yaml
processors:
resourcedetection:
detectors: [env, system, gcp, eks] # order = precedence
timeout: 2s
exporters:
otlphttp/metrics:
endpoint: "${env:METRICS_ENDPOINT}" # differs per environment, injected at runtime
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |