davila7/claude-code-templates / cli-tool/components/skills/ai-research/evaluation-nemo-evaluator
evaluation-nemo-evaluator
FlaggedEvaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.9 KB |
| references/adapter-system.md | markdown | 9.2 KB |
| references/configuration.md | markdown | 9.3 KB |
| references/custom-benchmarks.md | markdown | 6.9 KB |
| references/execution-backends.md | markdown | 9.1 KB |
references/adapter-system.md
View on GitHub# Adapter and Interceptor System
NeMo Evaluator uses an adapter system to process requests and responses between the evaluation engine and model endpoints. The `nemo-evaluator` core library provides built-in interceptors for common use cases.
## Architecture Overview
```
┌───────────────────────────────────────────────────────────────┐
│ Adapter Pipeline │
│ │
│ Request ───► [Interceptor 1] ───► [Interceptor 2] ───► │
│ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────┐ │
│ │ Endpoint Interceptor │ │
│ │ (HTTP call to Model API) │ │
│ └───────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Response ◄─── [Interceptor 3] ◄─── [Interceptor 4] ◄─── │
│ │
└───────────────────────────────────────────────────────────────┘
```
Interceptors execute in order for requests, and in reverse order for responses.
## Configuring Adapters
The adapter configuration is specified in the `target.api_endpoint.adapter_config` section:
```yaml
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY
adapter_config:
interceptors:
- name: system_message
config:
system_message: "You are a helpful assistant."
- name: caching
config:
cache_dir: "./cache"
- name: endpoint
- name: reasoning
config:
start_reasoning_token: "<think>"
end_reasoning_token: "</think>"
```
## Available Interceptors
### System Message Interceptor
Injects a system prompt into chat requests.
```yaml
- name: system_message
config:
system_message: "You are a helpful AI assistant. Think step by step."
```
**Effect**: Prepends a system message to the messages array.
### Request Logging Interceptor
Logs outbound API requests for debugging and analysis.
```yaml
- name: request_logging
config:
max_requests: 1000
```
### Caching Interceptor
Caches responses to avoid repeated API calls for identical requests.
```yaml
- name: caching
config:
cache_dir: "./evaluation_cache"
reuse_cached_responses: true
save_requests: true
save_responses: true
max_saved_requests: 1000
max_saved_responses: 1000
```
### Endpoint Interceptor
Performs the actual HTTP communication with the model endpoint. This is typically added automatically and has no configuration parameters.
```yaml
- name: endpoint
```
### Reasoning Interceptor
Extracts and removes reasoning tokens (e.g., `<think>` tags) from model responses.
```yaml
- name: reasoning
config:
start_reasoning_token: "<think>"
end_reasoning_token: "</think>"
enable_reasoning_tracking: true
```
**Effect**: Strips reasoning content from the response and tracks it separately.
### Response Logging Interceptor
Logs API responses.
```yaml
- name: response_logging
config:
max_responses: 1000
```
### Progress Tracking Interceptor
Reports evaluation progress to an external URL.
```yaml
- name: progress_tracking
config:
progress_tracking_url: "http://localhost:3828/progress"
progress_tracking_interval: 10
```
### Additional Interceptors
Other available interceptors include:
- `payload_modifier`: Transforms request parameters
- `response_stats`: Collects aggregated statistics from responses
- `raise_client_errors`: Handles and raises exceptions for client errors (4xx)
## Interceptor Chain Example
A typical interceptor chain for evaluation:
```yaml
adapter_config:
interceptors:
# Pre-endpoint (request processing)
- name: system_message
config:
system_message: "You are a helpful AI assistant."
- name: request_logging
config:
max_requests: 50
- name: caching
config:
cache_dir: "./evaluation_cache"
reuse_cached_responses: true
# Endpoint (HTTP call)
- name: endpoint
# Post-endpoint (response processing)
- name: response_logging
config:
max_responses: 50
- name: reasoning
config:
start_reasoning_token: "<think>"
end_reasoning_token: "</think>"
```
## Python API Usage
You can also configure adapters programmatically:
```python
from nemo_evaluator.adapters.adapter_config import AdapterConfig, InterceptorConfig
from nemo_evaluator.api.api_dataclasses import ApiEndpoint, EndpointType
adapter_config = AdapterConfig(
interceptors=[
InterceptorConfig(
name="system_message",
config={"system_message": "You are a helpful assistant."}
),
InterceptorConfig(
name="caching",
config={
"cache_dir": "./cache",
"reuse_cached_responses": True
}
),
InterceptorConfig(name="endpoint"),
InterceptorConfig(
name="reasoning",
config={
"start_reasoning_token": "<think>",
"end_reasoning_token": "</think>"
}
)
]
)
api_endpoint = ApiEndpoint(
url="http://localhost:8080/v1/chat/completions",
type=EndpointType.CHAT,
model_id="my_model",
adapter_config=adapter_config
)
```
## OpenAI API Compatibility
NeMo Evaluator supports OpenAI-compatible endpoints with different endpoint types:
### Chat Completions
```yaml
target:
api_endpoint:
type: chat # or omit, chat is default
url: http://endpoint/v1/chat/completions
```
### Text Completions
```yaml
target:
api_endpoint:
type: completions
url: http://endpoint/v1/completions
```
### Vision-Language Models
```yaml
target:
api_endpoint:
type: vlm
url: http://endpoint/v1/chat/completions
```
## Error Handling
Configure error handling via the `log_failed_requests` option:
```yaml
adapter_config:
log_failed_requests: true
interceptors:
- name: raise_client_errors
# ... other interceptors
```
## Debugging
### Enable Logging Interceptors
Add request and response logging to debug issues:
```yaml
adapter_config:
interceptors:
- name: request_logging
config:
max_requests: 100
- name: endpoint
- name: response_logging
config:
max_responses: 100
```
### Common Issues
**Issue: System message not applied**
Ensure the `system_message` interceptor is listed before the `endpoint` interceptor.
**Issue: Cache not being used**
Check that `reuse_cached_responses: true` is set and the cache directory exists:
```yaml
- name: caching
config:
cache_dir: "./cache"
reuse_cached_responses: true
```
**Issue: Reasoning tokens not extracted**
Verify the token patterns match your model's output format:
```yaml
- name: reasoning
config:
start_reasoning_token: "<think>" # Must match model output exactly
end_reasoning_token: "</think>"
```
## Custom Interceptor Discovery
NeMo Evaluator supports discovering custom interceptors via the `DiscoveryConfig` within `AdapterConfig`. You can specify modules or directories where your custom interceptors are located:
```yaml
adapter_config:
discovery:
modules:
- "my_custom.interceptors"
- "my_package.adapters"
dirs:
- "/path/to/custom/interceptors"
interceptors:
- name: my_custom_interceptor
config:
custom_option: value
```
Custom interceptors must implement the standard interceptor interface expected by `nemo-evaluator`.
## Additional AdapterConfig Options
Beyond interceptors, `AdapterConfig` supports these additional fields:
| Field | Description |
|-------|-------------|
| `discovery` | Configure custom interceptor discovery |
| `post_eval_hooks` | List of hooks to run after evaluation |
| `endpoint_type` | Default endpoint type (e.g., "chat") |
| `caching_dir` | Legacy option for response caching |
| `generate_html_report` | Generate HTML report of results |
| `log_failed_requests` | Log requests that fail |
| `tracking_requests_stats` | Enable request statistics |
| `html_report_size` | Number of request-response pairs in report |
## Notes
- The interceptor chain order matters - request interceptors run in order, response interceptors run in reverse
- Interceptors can be enabled/disabled via the `enabled` field in `InterceptorConfig`
- For complex custom logic, consider packaging as a custom container with your interceptors pre-installed
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |