m3taz-ahmed/aizee / skills/ai-observability-lord
ai-observability-lord
VerifiedLord skill for AI observability and tracing — OpenTelemetry + OpenInference standards, span attributes, eval integration, cost/latency/error tracking, drift detection, and PII redaction.
AI Observability Lord
[OBJ] Instrument AI systems with standards-based tracing, evaluation integration, cost/latency/error monitoring, and drift detection — with PII-safe traces and actionable dashboards.
Problem
AI systems are opaque: a request flows through prompt construction, retrieval, LLM call, tool use, guardrails, and response formatting — any of which can degrade silently. Traditional APM (latency + error rate) cannot see inside the LLM call. Without AI-specific observability, quality degrades undetected until users complain, and root cause is untraceable.
Rules
- [REQ] OpenTelemetry + OpenInference. Instrument all AI components with OpenTelemetry spans using OpenInference semantic conventions. Span attributes:
llm.model_name,llm.token_count.prompt,llm.token_count.completion,llm.tools,retrieval.documents,guardrail.verdict. No custom attribute names that duplicate OpenInference. - [REQ] Span attributes per component. LLM spans: model, tokens, temperature, system prompt hash. Tool spans: tool name, input, output, duration. Retrieval spans: query, documents retrieved, scores. Guardrail spans: verdict (allow/deny/redact), reason, latency. Every component type has a defined attribute set.
- [REQ] Tracing tool selection. LangSmith (LangChain ecosystem, managed), Langfuse v4 (open-source, self-hostable, multi-framework), Arize Phoenix (open-source, local-first, LLM + traditional ML), Braintrust (eval + observability), Helicone (proxy-based, OpenAI-focused), Portkey (gateway + observability). Match tool to stack and hosting preference.
- [REQ] Eval integration. Traces MUST link to evaluations. Every production trace can be scored by an evaluator (LLM-as-judge, rule-based, human). Eval scores appear as span attributes. Closed-loop: traces → evals → alerts → fixes → re-eval.
- [REQ] Cost tracking. Track cost per request: input tokens × price + output tokens × price + tool call costs + retrieval costs. Aggregate per user, per agent, per workflow. Cost anomaly (spike >3σ) triggers alert. Cost dashboard updated in real-time.
- [REQ] Latency monitoring. Track latency at each span: LLM call (TTFT + total), retrieval, tool execution, guardrail, end-to-end. P50/P95/P99 percentiles. SLO: P95 < target. Latency regression in CI = block.
- [REQ] Error tracking. Track errors by type: LLM error (rate limit, context overflow, content filter), tool error (timeout, auth, invalid input), retrieval error (no results, index down), guardrail error (misclassification). Error rate > threshold = alert with trace link.
- [REQ] Drift detection. Monitor for: input drift (prompt distribution shift), output drift (response distribution shift), performance drift (eval score decline). Use statistical tests (KS test, PSI) on rolling windows. Drift detected = alert + trigger eval re-run on recent traces.
- [REQ] Data residency. Traces contain user prompts and responses — PII. For EU/regulated deployments, self-host the tracing backend (Langfuse, Phoenix) in-region. No trace data leaves the jurisdiction. Document data residency per deployment.
- [REQ] Self-hosting vs cloud. Self-host (Langfuse, Phoenix) for: data residency, cost control at scale, air-gapped environments. Cloud (LangSmith, Braintrust) for: zero ops, fast setup, managed evals. Decision documented per project with rationale.
- [REQ] Retention policies. Define trace retention: 30 days for debugging, 90 days for trend analysis, 1 year for audit (sampled). Auto-delete expired traces. PII traces may have shorter retention. No indefinite retention without explicit policy.
- [REQ] Sampling strategies. Full tracing at low volume. At high volume: head-based sampling (sample by request attributes — always trace errors, always trace slow requests, sample 10% of normal). Tail-based sampling in OTel Collector. Never sample away all errors.
- [REQ] PII redaction in traces. Redact PII before storage: use regex + NER-based redaction on prompt and response text. Store redacted version in trace, original in encrypted vault with separate access control. No raw PII in trace storage.
- [REQ] Dashboard design. Dashboards show: request volume, latency percentiles, error rate, cost per request, eval score trend, drift indicators, top failing traces. Role-based: SRE sees ops metrics, ML sees quality metrics, product sees user-facing metrics.
- [REQ] Alerting. Alerts on: error rate > threshold, P95 latency > SLO, cost spike >3σ, eval score drop > threshold, drift detected. Alerts include trace link, affected user count, and suggested investigation path. No alert without a runbook.
- [PROHIBIT] Shipping an AI system to production without tracing, cost tracking, and PII redaction — untraced AI is unaccountable AI.
References
- OpenTelemetry: https://opentelemetry.io
- OpenInference: https://github.com/Arize-ai/openinference
- LangSmith: https://smith.langchain.com
- Langfuse: https://langfuse.com
- Arize Phoenix: https://phoenix.arize.com
- Braintrust: https://braintrust.dev
- Helicone: https://helicone.ai
- Portkey: https://portkey.ai
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |