respanai/respan / claude-plugin/skills/respan
respan
VerifiedRespan for LLM observability. Use to set up tracing or the gateway, manage prompts (create/version/deploy), run evals, experiments, and datasets, or work with traces and logs. Trigger for any Respan platform task or when instrumenting an app to send telemetry to Respan.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.5 KB |
| references/evals.md | markdown | 8.2 KB |
| references/gateway.md | markdown | 11.8 KB |
| references/monitors.md | markdown | 4.2 KB |
| references/prompts.md | markdown | 4.8 KB |
| references/tracing.md | markdown | 22.9 KB |
references/monitors.md
View on GitHub# Monitors & Automation Reference
Set up monitoring alerts, online evaluations, and automated workflows.
Docs: `https://respan.ai/docs/documentation/features/monitoring/monitors.md`
---
## Monitors
A monitor watches a metric over time and sends a notification when a threshold is crossed.
### Examples
- Error rate > 5% over 10 minutes
- Total cost > $1.00 over 5 minutes
- Average latency > 3 seconds over 15 minutes
- P95 latency > 5 seconds over 1 hour
### Setup
Monitors are configured in the platform UI: **Monitoring -> Monitors -> Create**.
A monitor has two parts:
#### 1. Trigger
Template: `When [metric] of [source] is [threshold] over [time window]`
- **Metric:** cost, latency, error rate, token usage, request count, etc.
- **Source:** all requests, specific model, specific customer, etc.
- **Threshold:** condition (greater than, less than, equals)
- **Time window:** evaluation period (1 min, 5 min, 10 min, 1 hour, etc.)
Add **Where** conditions to narrow scope:
- Filter by model, project, environment, customer, or custom metadata
#### 2. Notifications
Destinations:
| Destination | Details |
|-------------|---------|
| **Email** | Customizable subject and body |
| **Slack** | Customizable message |
| **Webhook** | Raw URL — receives JSON payload |
A single monitor can send to multiple destinations.
### Manage
- **Deploy** to make a monitor live
- **Pause/Resume** deployed monitors
- **Send test alert** to validate before deploying
- **View versions** and restore earlier versions as drafts
- **Create from chart** — select a metric on the dashboard, click to create a monitor with metric/source/interval prefilled
---
## Online Evaluations (Automation)
Automatically score production traffic in real time using deployed evaluators.
### Setup
Configure in the platform UI: **Evaluators -> Automations -> Create**.
1. **Create a condition** — filter rules for which logs to evaluate:
- **Single log** — evaluate individual spans matching criteria
- **Aggregation** — evaluate aggregated metrics over a window
2. **Create an automation** — connect:
- Condition (which logs)
- Evaluator (which scoring workflow)
- Sampling rate (0.0-1.0, fraction of matching logs to evaluate)
3. **Check results** — view eval scores in the automation table
### Use case
> "Score 10% of production chat responses for hallucination using my LLM evaluator"
```
Condition: log_type = "chat" AND model = "gpt-4o"
Evaluator: hallucination-check
Sampling: 0.1
```
---
## Webhooks
Receive event notifications via HTTP webhooks.
### Setup
1. Go to platform: **Settings -> Webhooks -> Create**
2. Enter your endpoint URL
3. Select event types to subscribe to
4. Optionally set a webhook secret for HMAC-SHA256 verification
### Verification
Respan signs webhook payloads with HMAC-SHA256. Verify the signature:
```python
import hashlib
import hmac
def verify_webhook(payload: bytes, signature: str, secret: str) -> bool:
expected = hmac.new(
secret.encode(),
payload,
hashlib.sha256,
).hexdigest()
return hmac.compare_digest(expected, signature)
```
The signature is sent in the request headers.
---
## Metrics Dashboard
Track key indicators in the platform: **Dashboard -> Metrics**.
### Available metrics
- **Requests:** total count, requests per model/customer/API key
- **Tokens:** prompt tokens, completion tokens, total tokens
- **Cost:** per-request, per-model, per-customer
- **Latency:** average, P50, P95, P99
- **Errors:** error count, error rate
- **Time to first token (TTFT):** streaming latency
### Breakdowns
Slice metrics by:
- Model
- Customer (`customer_identifier`)
- API key
- Prompt template
- Environment
- Custom metadata keys
### Per-user analytics
Use `customer_identifier` in spans to enable:
- Per-user cost tracking and budgets
- Per-user rate limiting
- User-level quality metrics
---
## CLI Commands
```bash
respan traces list --limit 10 # List recent traces
respan traces get <trace-id> # Get trace details
respan traces summary # Trace summary stats
respan logs list --limit 10 # List recent logs
respan logs get <log-id> # Get log details
respan logs summary # Log summary stats
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |