davila7/claude-code-templates / cli-tool/components/skills/ai-research/evaluation-nemo-evaluator
evaluation-nemo-evaluator
FlaggedEvaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.9 KB |
| references/adapter-system.md | markdown | 9.2 KB |
| references/configuration.md | markdown | 9.3 KB |
| references/custom-benchmarks.md | markdown | 6.9 KB |
| references/execution-backends.md | markdown | 9.1 KB |
references/configuration.md
View on GitHub# Configuration Reference
NeMo Evaluator uses Hydra for configuration management with a hierarchical override system.
## Configuration Structure
```yaml
# Complete configuration structure
defaults:
- execution: local # Execution backend
- deployment: none # Model deployment method
- _self_
execution:
# Executor-specific settings
output_dir: ./results
mode: sequential
target:
# Model endpoint settings
api_endpoint:
model_id: model-name
url: http://endpoint/v1/chat/completions
api_key_name: API_KEY
type: chat # chat, completions, vlm, embedding
adapter_config:
interceptors: []
evaluation:
# Global evaluation settings
nemo_evaluator_config:
config:
params:
temperature: 0.0
parallelism: 4
# Task list
tasks:
- name: task_name
env_vars: {}
nemo_evaluator_config: {} # Per-task overrides
```
## Configuration Sections
### Defaults Section
Selects base configurations for execution and deployment:
```yaml
defaults:
- execution: local # Options: local, slurm, lepton
- deployment: none # Options: none, vllm, sglang, nim
- _self_
```
Available execution configs:
- `local` - Docker-based local execution
- `slurm` - HPC cluster via SSH/sbatch
- `lepton` - Lepton AI cloud platform
Available deployment configs:
- `none` - Evaluate existing endpoint
- `vllm` - Deploy model with vLLM
- `sglang` - Deploy model with SGLang
- `nim` - Deploy model with NVIDIA NIM
### Execution Section
Controls how and where evaluations run:
```yaml
execution:
# Common settings
output_dir: ./results # Where to write results
mode: sequential # sequential or parallel
# Local executor specific
docker_args:
- "--gpus=all"
- "--shm-size=16g"
memory_limit: "64g"
cpus: 8
# Slurm executor specific
hostname: cluster.example.com
account: my_account
partition: gpu
qos: normal
nodes: 1
gpus_per_node: 8
walltime: "04:00:00"
# Lepton executor specific
resource_shape: gpu.a100-80g
num_replicas: 1
```
### Target Section
Specifies the model endpoint to evaluate:
```yaml
target:
api_endpoint:
# Required fields
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY # Environment variable name
# Optional fields
type: chat # chat, completions, vlm, embedding
timeout: 300 # Request timeout in seconds
max_retries: 3 # Retry count for failed requests
# Adapter configuration
adapter_config:
interceptors:
- name: system_message
config:
system_message: "You are a helpful assistant."
- name: caching
config:
cache_dir: "./cache"
- name: reasoning
config:
start_reasoning_token: "<think>"
end_reasoning_token: "</think>"
```
### Evaluation Section
Configures tasks and evaluation parameters:
```yaml
evaluation:
# Global parameters (apply to all tasks)
nemo_evaluator_config:
config:
params:
temperature: 0.0 # Sampling temperature
max_new_tokens: 512 # Max generation length
parallelism: 4 # Concurrent requests
limit_samples: null # Limit samples (null = all)
num_fewshot: 5 # Few-shot examples
random_seed: 42 # Random seed
# Task list
tasks:
- name: ifeval
- name: gpqa_diamond
env_vars:
HF_TOKEN: HF_TOKEN # Task-specific env vars
- name: gsm8k_cot_instruct
nemo_evaluator_config: # Task-specific overrides
config:
params:
temperature: 0.0
max_new_tokens: 1024
```
## Configuration Override Precedence
Configurations are resolved in this order (highest to lowest):
1. **CLI overrides**: `-o key=value`
2. **Task-specific** `nemo_evaluator_config`
3. **Global** `evaluation.nemo_evaluator_config`
4. **Framework defaults** (in container)
5. **System defaults**
## CLI Override Syntax
### Basic Overrides
```bash
# Override simple values
-o execution.output_dir=/custom/path
-o target.api_endpoint.model_id=my-model
# Override nested values
-o target.api_endpoint.adapter_config.interceptors[0].name=logging
```
### Adding New Values
Use `+` prefix to add values not in base config:
```bash
# Add evaluation parameter
-o +evaluation.nemo_evaluator_config.config.params.limit_samples=100
# Add environment variable
-o +target.api_endpoint.env_vars.CUSTOM_VAR=value
```
### Complex Values
```bash
# Override list/array
-o 'evaluation.tasks=[{name: ifeval}, {name: gsm8k}]'
# Override with special characters (use quotes)
-o 'target.api_endpoint.url="http://localhost:8000/v1/chat/completions"'
```
### Multi-Override
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name config \
-o execution.output_dir=/results \
-o target.api_endpoint.model_id=my-model \
-o +evaluation.nemo_evaluator_config.config.params.parallelism=8 \
-o +evaluation.nemo_evaluator_config.config.params.limit_samples=10
```
## Task Configuration
### Task Discovery
List available tasks and their requirements:
```bash
# List all tasks
nemo-evaluator-launcher ls tasks
# Output includes:
# - Task name
# - Container image
# - Endpoint type (chat/completions/vlm)
# - Required environment variables
```
### Task Environment Variables
Some tasks require specific environment variables:
| Task | Required Env Vars |
|------|------------------|
| `gpqa_diamond` | `HF_TOKEN` |
| `mmlu` | `HF_TOKEN` |
| `math_test_500_nemo` | `JUDGE_API_KEY` |
| `aime` | `JUDGE_API_KEY` |
| `slidevqa` | `OPENAI_CLIENT_ID`, `OPENAI_CLIENT_SECRET` |
Configure in task definition:
```yaml
evaluation:
tasks:
- name: gpqa_diamond
env_vars:
HF_TOKEN: HF_TOKEN # Maps to $HF_TOKEN from environment
- name: math_test_500_nemo
env_vars:
JUDGE_API_KEY: MY_JUDGE_KEY # Maps to $MY_JUDGE_KEY
```
### Task-Specific Parameters
Override parameters for specific tasks:
```yaml
evaluation:
nemo_evaluator_config:
config:
params:
temperature: 0.0 # Global default
tasks:
- name: ifeval
# Uses global temperature: 0.0
- name: humaneval
nemo_evaluator_config:
config:
params:
temperature: 0.8 # Override for code generation
max_new_tokens: 1024
n_samples: 200 # Multiple samples for pass@k
```
## Adapter Configuration
Adapters intercept and process requests/responses:
```yaml
target:
api_endpoint:
adapter_config:
# Request interceptors (before API call)
interceptors:
- name: system_message
config:
system_message: "You are a helpful assistant."
- name: request_logging
config:
max_logged_requests: 100
log_path: "./logs/requests.jsonl"
- name: caching
config:
cache_dir: "./cache"
cache_ttl: 3600
# Response interceptors (after API call)
- name: reasoning
config:
start_reasoning_token: "<think>"
end_reasoning_token: "</think>"
strip_reasoning: true
- name: response_logging
config:
max_logged_responses: 100
# Error handling
log_failed_requests: true
retry_on_failure: true
max_retries: 3
```
## Example Configurations
### Minimal Local Evaluation
```yaml
defaults:
- execution: local
- deployment: none
- _self_
execution:
output_dir: ./results
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY
evaluation:
tasks:
- name: ifeval
```
### Production Slurm Evaluation
```yaml
defaults:
- execution: slurm
- deployment: vllm
- _self_
execution:
hostname: cluster.example.com
account: research_account
partition: gpu
nodes: 2
gpus_per_node: 8
walltime: "08:00:00"
output_dir: /shared/results/$(date +%Y%m%d)
deployment:
checkpoint_path: /models/llama-3.1-70b
tensor_parallel_size: 8
data_parallel_size: 2
max_model_len: 8192
evaluation:
nemo_evaluator_config:
config:
params:
parallelism: 16
temperature: 0.0
tasks:
- name: mmlu_pro
- name: gsm8k_cot_instruct
- name: ifeval
- name: gpqa_diamond
env_vars:
HF_TOKEN: HF_TOKEN
```
### Quick Testing Configuration
```yaml
defaults:
- execution: local
- deployment: none
- _self_
execution:
output_dir: ./test_results
target:
api_endpoint:
model_id: meta/llama-3.1-8b-instruct
url: https://integrate.api.nvidia.com/v1/chat/completions
api_key_name: NGC_API_KEY
evaluation:
nemo_evaluator_config:
config:
params:
limit_samples: 10 # Only 10 samples per task
parallelism: 2
tasks:
- name: ifeval
- name: gsm8k_cot_instruct
```
## Validation
### Dry Run
Validate configuration without execution:
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name config \
--dry-run
```
### Common Validation Errors
**Missing required field**:
```
ValidationError: target.api_endpoint.model_id is required
```
**Invalid task name**:
```
TaskNotFoundError: Task 'invalid_task' not found in mapping.toml
```
**Missing environment variable**:
```
EnvVarError: Task 'gpqa_diamond' requires HF_TOKEN but it is not set
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |