NVIDIA/SkillEvaluator / src/skillevaluator/tier3/reference_skills/create-custom-grader
create-custom-grader
VerifiedUse when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
evals/evals.json
View on GitHub{
"skill_name": "create-custom-grader",
"evals": [
{
"id": "create-custom-grader-001",
"expected_skill": "create-custom-grader",
"expected_script": null,
"prompt": "The benchmark folder is input/create-custom-grader-001/benchmark/. It has task.yaml, code/ fixtures, prompt variants, coverage.target_skill, expected non-trigger skills, and a five-item rubric. Convert it into an SkillEvaluator bring-your-own-grader workflow for the target skill, using default evaluator metrics plus custom rubric metrics.",
"expected_output": "The agent uses create-custom-grader to map the benchmark into SkillEvaluator BYOG: evals/evals.json entries from the benchmark prompts, fixtures under evals/files, evals/config.yml with grading.mode default_plus_custom, a real evals/grader.py or grader.sh that writes reward.json/reward.txt with custom_metrics for the rubric, and validation/evaluation commands.",
"assertions": [
"The agent reads create-custom-grader/SKILL.md before designing the conversion.",
"The agent chooses top-level BYOG with grading.mode default_plus_custom unless the user asks to replace default metrics.",
"The agent maps benchmark prompts, target skill, trigger boundaries, fixtures, and rubric criteria into evaluator files instead of storing all benchmark metadata blindly in evals.json.",
"The agent implements executable grader logic that writes /logs/verifier/reward.json and /logs/verifier/reward.txt with numeric custom metrics.",
"The agent runs or explicitly plans skillevaluator validate --harbor-contract and reports what was proven."
],
"files": [
"files/create-custom-grader-001"
]
},
{
"id": "create-custom-grader-002",
"expected_skill": "create-custom-grader",
"expected_script": null,
"prompt": "Our skill at input/create-custom-grader-002/csv-summary/ already has evals/evals.json, but we want to add one more verifier that checks generated files and reports a domain-specific score alongside the standard evaluator metrics. Build the custom grader structure and explain the data exchange contract.",
"expected_output": "The agent adds a top-level BYOG grader for the existing generated tasks, keeps grading.mode default_plus_custom, explains that the grader reads /logs/agent/trajectory.json, /tests/entry.json, and workspace artifacts, and writes reward.json plus reward.txt with non-reserved custom_metrics. It preserves the existing evals.json tasks and validates the Harbor contract.",
"assertions": [
"The agent reads create-custom-grader/SKILL.md before acting.",
"The agent keeps the existing evals/evals.json as the task source rather than converting to native Harbor unnecessarily.",
"The agent uses or recommends skillevaluator init-custom-grader with the appropriate language.",
"The agent avoids reserved metric-name collisions and keeps default evaluator metrics separate from custom metrics.",
"The agent includes validation and E2E verification steps rather than stopping at placeholder files."
],
"files": [
"files/create-custom-grader-002"
]
},
{
"id": "create-custom-grader-003-negative",
"expected_skill": null,
"expected_script": null,
"should_trigger": false,
"prompt": "Create a normal three-case evals/evals.json dataset for a calculator skill. I do not need a custom grader or any Harbor task files.",
"expected_output": "The agent recognizes this as ordinary eval dataset authoring and does not activate create-custom-grader. It uses the normal dataset workflow instead and does not create grader.py, grader.sh, evals/harbor, or custom grading config.",
"assertions": [
"The agent does not read or activate create-custom-grader/SKILL.md.",
"The agent does not create custom grader files when the user explicitly says they are not needed.",
"The agent routes to normal evals/evals.json dataset authoring."
],
"files": []
}
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |