comet-ml/opik-skills / skills/opik
opik
VerifiedReference for the Opik SDK — tracing, span types, framework integrations, threads, and the prompt library (Python, TypeScript, REST). Use for "what span types exist", "how do I flush", "track_openai", "add OpikTracer", "version a prompt". To instrument a repo end to end, use the `opik-instrument` skill.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.3 KB |
| references/agent-patterns.md | markdown | 17.1 KB |
| references/best-practices.md | markdown | 4.8 KB |
| references/evaluation-datasets.md | markdown | 16.9 KB |
| references/evaluation-test-suites.md | markdown | 16.8 KB |
| references/integrations.md | markdown | 15.1 KB |
| references/observability.md | markdown | 2.3 KB |
| references/production.md | markdown | 11.4 KB |
| references/tracing-python.md | markdown | 23.0 KB |
| references/tracing-rest-api.md | markdown | 9.3 KB |
| references/tracing-typescript.md | markdown | 10.5 KB |
references/best-practices.md
View on GitHub# Best Practices: Build, Evaluate, and Monitor AI Agents
Best practices for the agent lifecycle beyond basic tracing — architecture patterns, evaluation, metrics, and production monitoring. This is reference material for the `opik` skill. For SDK details (tracing, integrations, span types) see the other `opik` references; to actually build and run an evaluation, use the `opik-evaluate` skill.
## The Agent Lifecycle
1. **Instrument** — Add Opik tracing to make your agent's behavior visible (see `opik` skill)
2. **Evaluate** — Measure performance with datasets, metrics, and experiments
3. **Monitor** — Track quality, cost, and reliability in production
4. **Optimize** — Improve based on data from evaluation and production traces
## Agent Architecture Patterns
Trace every component of your agent with appropriate span types:
```python
import opik
@opik.track(name="research_agent")
def agent(query: str) -> str:
plan = plan_action(query) # general span
results = execute_tool(plan) # tool span
return generate_response(results) # llm span
@opik.track(type="tool")
def execute_tool(action: dict) -> str:
return search_web(action["query"])
@opik.track(type="llm")
def generate_response(context: str) -> str:
return llm_call(context)
```
### What to Trace
| Component | Span Type | Key Data |
|-----------|-----------|----------|
| Planning | `general` | Reasoning steps, decisions |
| Tool calls | `tool` | Tool name, parameters, results |
| LLM calls | `llm` | Prompt, response, tokens |
| Retrieval | `tool` | Query, documents |
| Validation | `guardrail` | Check results, pass/fail |
## Evaluation
### Test Suites (Recommended)
Test Suites with `run_tests()` / `runTests()` are the recommended way to evaluate agents. Assertions are plain strings checked by an LLM judge, with execution policies for multi-run reliability:
**Python:**
```python
import opik
client = opik.Opik()
suite = client.get_or_create_test_suite(
name="my-agent-suite",
global_assertions=[
"Response is factually accurate and not hallucinated",
"Response is professional in tone",
],
global_execution_policy={"runs_per_item": 3, "pass_threshold": 2},
)
results = opik.run_tests(
test_suite=suite,
task=lambda item: {"output": agent(item["input"])},
)
assert results.all_items_passed # CI gate
```
**TypeScript:**
```typescript
import { Opik, runTests } from "opik";
const client = new Opik();
const suite = await client.getOrCreateTestSuite({
name: "my-agent-suite",
globalAssertions: [
"Response is factually accurate and not hallucinated",
"Response is professional in tone",
],
globalExecutionPolicy: { runsPerItem: 3, passThreshold: 2 },
});
const results = await runTests({
testSuite: suite,
task: async (item) => ({
input: item.input,
output: await agent(item.input as string),
}),
});
if (!results.allItemsPassed) process.exit(1); // CI gate
```
See `evaluation-test-suites.md` for full API (items, versioning, execution policies, CI integration).
### Legacy Dataset Evaluation
For custom metric-based evaluation (existing workflows or explicit user request), use `evaluate()` with datasets:
```python
from opik.evaluation import evaluate
from opik.evaluation.metrics import AnswerRelevance, Hallucination, AgentTaskCompletionJudge
results = evaluate(
experiment_name="agent-v2",
dataset=dataset,
task=lambda item: {"output": agent(item["input"])},
scoring_metrics=[
AnswerRelevance(),
Hallucination(),
AgentTaskCompletionJudge(),
]
)
```
See `evaluation-datasets.md` for full API (datasets, versioning, 60+ metrics, annotation queues).
## Production Monitoring
- **Dashboards** — Visualize quality, cost, latency, and error trends
- **Online evaluation** — Automatically score production traces with LLM-as-Judge
- **Alerts** — Get notified when metrics deviate (quality drops, cost spikes, error rates)
- **Guardrails** — PII detection, topic validation, custom safety checks
- **Ollie** — AI-powered root cause analysis for failed traces (Opik's AI assistant)
## Common Anti-Patterns
| Category | Anti-Pattern |
|----------|-------------|
| Reliability | Unbounded loops, retry storms, silent failures |
| Security | Prompt injection, privilege escalation, data leakage |
| Observability | Late tracing (missing input), orphaned spans |
| Tools | Tool loops, hallucinated tools, parameter errors |
## Detailed References
| Topic | Reference File |
|-------|----------------|
| Agent architecture, reliability, security patterns | `agent-patterns.md` |
| **Test Suites, `run_tests()`, assertions, CI gating (recommended)** | `evaluation-test-suites.md` |
| Legacy datasets, `evaluate()`, 60+ metrics, annotation queues | `evaluation-datasets.md` |
| Production dashboards, alerts, guardrails, cost tracking | `production.md` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |