davila7/claude-code-templates / cli-tool/components/skills/ai-research/evaluation-nemo-evaluator
evaluation-nemo-evaluator
FlaggedEvaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.9 KB |
| references/adapter-system.md | markdown | 9.2 KB |
| references/configuration.md | markdown | 9.3 KB |
| references/custom-benchmarks.md | markdown | 6.9 KB |
| references/execution-backends.md | markdown | 9.1 KB |
references/custom-benchmarks.md
View on GitHub# Custom Benchmark Integration
NeMo Evaluator supports adding custom benchmarks through Framework Definition Files (FDFs) and custom containers.
## Overview
Custom benchmarks are added by:
1. **Framework Definition Files (FDFs)**: YAML files that define evaluation tasks, commands, and output parsing
2. **Custom Containers**: Package your framework with nemo-evaluator for reproducible execution
> **Note**: NeMo Evaluator does not currently support programmatic harness APIs or custom metric implementations via Python classes. Customization is done through FDFs and containers.
## Framework Definition Files (FDFs)
FDFs are the primary way to add custom evaluations. An FDF declares framework metadata, default commands, and evaluation tasks.
### FDF Structure
```yaml
# framework_def.yaml
framework:
name: my-custom-framework
package_name: my_custom_eval
defaults:
command: "python -m my_custom_eval.run --model-id {model_id} --task {task} --output-dir {output_dir}"
evaluations:
- name: custom_task_1
defaults:
temperature: 0.0
max_new_tokens: 512
extra:
custom_param: value
- name: custom_task_2
defaults:
temperature: 0.7
max_new_tokens: 1024
```
### Key FDF Components
**Framework section**:
- `name`: Human-readable name for your framework
- `package_name`: Python package name
**Defaults section**:
- `command`: The command template to execute your evaluation
- Placeholders: `{model_id}`, `{task}`, `{output_dir}` are substituted at runtime
**Evaluations section**:
- List of tasks with their default parameters
- Each task can override the framework defaults
### Output Parser
When creating a custom FDF, you need an output parser function that translates your framework's results into NeMo Evaluator's standard schema:
```python
# my_custom_eval/parser.py
def parse_output(output_dir: str) -> dict:
"""
Parse evaluation results from output_dir.
Returns dict with metrics in NeMo Evaluator format.
"""
# Read your framework's output files
results_file = Path(output_dir) / "results.json"
with open(results_file) as f:
raw_results = json.load(f)
# Transform to standard schema
return {
"metrics": {
"accuracy": raw_results["score"],
"total_samples": raw_results["num_samples"]
}
}
```
## Custom Container Creation
Package your custom framework as a container for reproducibility.
### Dockerfile Example
```dockerfile
# Dockerfile
FROM python:3.10-slim
# Install nemo-evaluator
RUN pip install nemo-evaluator
# Install your custom framework
COPY my_custom_eval/ /opt/my_custom_eval/
RUN pip install /opt/my_custom_eval/
# Copy framework definition
COPY framework_def.yaml /opt/framework_def.yaml
# Set working directory
WORKDIR /opt
ENTRYPOINT ["python", "-m", "nemo_evaluator"]
```
### Build and Push
```bash
docker build -t my-registry/custom-eval:1.0 .
docker push my-registry/custom-eval:1.0
```
### Register in mapping.toml
Add your custom container to the task registry:
```toml
# Add to mapping.toml
[my-custom-framework]
container = "my-registry/custom-eval:1.0"
[my-custom-framework.tasks.chat.custom_task_1]
required_env_vars = []
[my-custom-framework.tasks.chat.custom_task_2]
required_env_vars = ["CUSTOM_API_KEY"]
```
## Using Custom Datasets
### Dataset Mounting
Mount proprietary datasets at runtime rather than baking them into containers:
```yaml
# config.yaml
defaults:
- execution: local
- deployment: none
- _self_
execution:
output_dir: ./results
evaluation:
tasks:
- name: custom_task_1
dataset_dir: /path/to/local/data
dataset_mount_path: /data # Optional, defaults to /datasets
```
The launcher will mount the dataset directory into the container and set `NEMO_EVALUATOR_DATASET_DIR` environment variable.
### Task-Specific Environment Variables
Pass environment variables to specific tasks:
```yaml
evaluation:
tasks:
- name: gpqa_diamond
env_vars:
HF_TOKEN: HF_TOKEN # Maps to $HF_TOKEN from host
- name: custom_task
env_vars:
CUSTOM_API_KEY: MY_CUSTOM_KEY
DATA_PATH: /data/custom.jsonl
```
## Parameter Overrides
Override evaluation parameters at multiple levels:
### Global Overrides
Apply to all tasks:
```yaml
evaluation:
nemo_evaluator_config:
config:
params:
temperature: 0.0
max_new_tokens: 512
parallelism: 4
request_timeout: 300
```
### Task-Specific Overrides
Override for individual tasks:
```yaml
evaluation:
tasks:
- name: humaneval
nemo_evaluator_config:
config:
params:
temperature: 0.8
max_new_tokens: 1024
n_samples: 200 # Task-specific parameter
```
### CLI Overrides
Override at runtime:
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name config \
-o +evaluation.nemo_evaluator_config.config.params.limit_samples=10
```
## Testing Custom Benchmarks
### Dry Run
Validate configuration without execution:
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name custom_config \
--dry-run
```
### Limited Sample Testing
Test with a small subset first:
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name custom_config \
-o +evaluation.nemo_evaluator_config.config.params.limit_samples=5
```
### Check Results
```bash
# View results
cat results/<invocation_id>/<task>/artifacts/results.json
# Check logs
cat results/<invocation_id>/<task>/artifacts/logs/eval.log
```
## Best Practices
1. **Use FDFs**: Define custom benchmarks via Framework Definition Files
2. **Containerize**: Package frameworks as containers for reproducibility
3. **Mount data**: Use volume mounts for datasets instead of baking into images
4. **Test incrementally**: Use `limit_samples` for quick validation
5. **Version containers**: Tag containers with semantic versions
6. **Document parameters**: Include clear documentation in your FDF
## Limitations
Currently **not supported**:
- Custom Python metric classes via plugin system
- Programmatic harness registration via Python API
- Runtime metric injection via configuration
Custom scoring logic must be implemented within your evaluation framework and exposed through the FDF's output parser.
## Example: Complete Custom Setup
```yaml
# custom_eval_config.yaml
defaults:
- execution: local
- deployment: none
- _self_
execution:
output_dir: ./custom_results
target:
api_endpoint:
model_id: my-model
url: http://localhost:8000/v1/chat/completions
api_key_name: ""
evaluation:
nemo_evaluator_config:
config:
params:
parallelism: 4
request_timeout: 300
tasks:
- name: custom_task_1
dataset_dir: /data/benchmarks
env_vars:
DATA_VERSION: v2
nemo_evaluator_config:
config:
params:
temperature: 0.0
max_new_tokens: 256
```
Run with:
```bash
nemo-evaluator-launcher run \
--config-dir . \
--config-name custom_eval_config
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |