JPeetz/agent-skills / skills/devops-infrastructure/llm-observability
llm-observability
VerifiedUse this skill when instrumenting, tracing, evaluating, or monitoring LLM and agent applications — capturing prompts and tool calls as traces, tracking token cost and latency, running LLM-as-judge evaluations, building quality dashboards and alerts, or gating model/prompt changes with regression tests. Implements OpenTelemetry GenAI conventions with Langfuse, Phoenix, LangSmith, Opik, or Weave backends.
| File | Kind | Size |
|---|---|---|
| CHANGELOG.md | markdown | 2.3 KB |
| LICENSE | other | 1.0 KB |
| SKILL.md | skill md | 26.3 KB |
| evals/evals.json | config | 3.5 KB |
| references/evaluation-playbook.md | markdown | 5.5 KB |
| references/otel-genai-attributes.md | markdown | 6.6 KB |
| references/platform-selection.md | markdown | 4.3 KB |
| scripts/check_genai_attrs.pyexecutable | script | 7.3 KB |
| scripts/langfuse-compose.shexecutable | script | 4.7 KB |
CHANGELOG.md
View on GitHub# Changelog
All notable changes to the llm-observability skill.
## v1.0.0 — 2026-11-14
### Initial Release
Production observability for generative-AI systems: tracing, evaluation, cost
attribution, quality dashboards, and CI regression gates.
### What's Included
**Core Capabilities**
- Five-phase workflow: backend selection → instrumentation → evaluation →
dashboards/alerts → CI regression gates
- OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes, span naming,
`CLIENT` span kind) with development-status caveats and version-pinning advice
- Backend comparison: Langfuse, Arize Phoenix, LangSmith, Opik, Weave, plain OTLP
- Collector-based pipeline: content redaction, cost derivation, tail sampling,
dual-export during migration
- Evaluator selection matrix (deterministic checks first, judges only where code
cannot express the criterion)
- Judge calibration protocol: hand labels, ≥0.8 agreement, `temperature=0`,
different model family from the system under test, structured verdicts
- Online (production sampling + user feedback) and offline (dataset) evaluation
- Six-panel dashboard spec and a rate/delta-based alert table with actions
- Aggregated-delta quality gates in CI with baselines committed to the repo
- Boundary section defining the split from `observability-engineering`:
service SLOs and RED/USE metrics there, prompt/token/quality semantics here,
with nested traces across the two
**References**
- `references/platform-selection.md` — decision matrix, self-host sizing, migration paths
- `references/otel-genai-attributes.md` — attribute and metric reference, drift notes
- `references/evaluation-playbook.md` — judge design, calibration, dataset curation
**Scripts**
- `scripts/langfuse-compose.sh` — bring up a local Langfuse stack (`up`/`smoke`/`down`)
- `scripts/check_genai_attrs.py` — lint exported OTLP/JSON traces for missing GenAI attributes
**Evals**
- 6 cases: 4 trigger cases (Langfuse instrumentation, OTLP export to an existing
Collector, prompt-change quality gate, cost-spike investigation) and 2 near-miss
negatives (plain service monitoring, prompt writing without measurement intent)
### Licence
MIT. Note that Arize Phoenix itself ships under ELv2; the licence here covers
this skill's documentation and scripts only.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |