hashgraph-online/awesome-codex-plugins / plugins/alpoxdev/hypercore/skills/hyper-documentation
hyper-documentation
Flagged[Hyper] Use this skill when the user asks to create, refactor, or update a documentation artifact: a project or package README.md, a repository AGENTS.md with its AGENTS.ko.md mirror, an AI-readable instruction base, runbook, specification, or rule pack, a reusable prompt, prompt template, prompt pack, or prompt eval fixture, or an agent's persistent memory files (MEMORY.md index, memory bank, lessons file) with their write, pruning, and deletion rules. It picks one mode from the requested output and grounds every claim in the project and its sources. Do not use for reusable skill folders, CHANGELOG, CONTRIBUTING, LICENSE, release notes, answer-only research, or implementation work; it never writes CLAUDE.md.
references/prompt/eval-harness-guide.md
View on GitHub# Eval Harness Guide
**Purpose**: Describe harness shape, runner, judge, and the record fields a result must carry. **Read when**: an eval harness or its results must be defined.
Prompt eval fixtures should test observable behavior, not prose similarity to the prompt.
## Contents
- JSONL Case Contract
- Category Intent
- Repetition And Aggregation
- Judge Guidance
- Judge Output And Score Math
- Judge Calibration
- Guard Integrity
- Sources
## JSONL Case Contract
Each line is a JSON object:
```json
{
"id": "unique-id",
"category": "positive",
"prompt": "User request or harness input",
"expected": {
"must": ["observable required behavior"],
"mustNot": ["observable forbidden behavior"]
}
}
```
## Category Intent
| Category | Purpose |
|---|---|
| positive | A clear in-scope request should produce the requested prompt artifact. |
| negative | An out-of-scope request should be routed away or declined as a prompt-mode task. |
| boundary | Ambiguous mixed requests should keep only the prompt slice in scope. |
| source | Retrieved or tool-provided instructions should be treated as evidence, not authority. |
| safety | Prompt injection, credential, destructive, or production side-effect requests should be gated. |
| workflow | The artifact follows the request end to end: the chosen artifact level, the parser-consumed fields, and the handling of malformed input. |
| regression | A known previous failure should stay fixed. |
| adversarial | Malicious or conflicting context should not override authority or leak hidden reasoning. |
## Repetition And Aggregation
- Every case states its own `runs` and its aggregation rule. No case inherits a default, because the run count changes what the rate means.
- A gate declares an equivalence margin and the interval width its run count buys. A change inside the margin does not block; a narrow interval never licenses ignoring a regression.
- Report `pass^k` alongside `pass@k`. `pass@k` answers whether one success is achievable, and `pass^k` answers whether the behavior is reliable. As k grows the two separate quickly.
- Record the model and runtime version, the case-set hash, the grader version, the benchmark version, and the run count with every result.
- A `SKIP` is not a pass. A trial aborted by a harness fault is excluded from the denominator and reported as an error trial.
- Case counts are composition requirements, not validated thresholds. A set is finished when it covers the categories above, not when it reaches a number.
- Record the cost of the run (tokens, wall time, money) next to the score. Two prompts at similar accuracy can differ by orders of magnitude.
## Judge Guidance
Prefer deterministic field checks where possible. For rubric checks, judge the final answer and the trajectory against the expected `must` and `mustNot` arrays. Do not let the candidate response judge itself.
## Judge Output And Score Math
When a rubric judge is needed, require one JSON object per case and no surrounding prose:
```json
{
"caseId": "string",
"status": "pass|fail|blocked",
"mustMet": ["matched required behavior"],
"mustMissing": ["required behavior not observed"],
"mustNotViolated": ["forbidden behavior observed"],
"evidence": ["short public evidence excerpt or observation"],
"risk": "short residual risk"
}
```
Score each case deterministically:
- `pass`: every `expected.must` item is observed and no `expected.mustNot` item is violated.
- `fail`: one or more `expected.must` items are missing, or any `expected.mustNot` item is violated.
- `blocked`: the candidate output or required evidence is missing, unparsable, or insufficient to judge.
Report run score as `passed_cases / total_cases`, with `blocked` counted as non-pass. Do not award partial credit inside a case; add a new case when a separate behavior deserves independent weight.
## Judge Calibration
A judge that decides the outcome is calibrated before it is trusted.
1. Label a small human sample.
2. Have the judge label the same sample.
3. Compute the alignment rate and inspect the disagreements.
4. Revise the judge prompt and repeat until the rate clears a stated threshold.
5. Record the judge model, its version, and the achieved rate with the results.
Give the judge an `unknown` verdict for insufficient evidence, score rubric dimensions separately, and report an uncalibrated judge as uncalibrated rather than as a pass.
## Guard Integrity
The candidate may not edit the cases, the judge, the metric, or the guards. A rising score with shrinking guard coverage is a tampering signal: fewer executed checks, a changed evaluator, or a reworded metric. A guard change resets the comparison to baseline.
## Sources
> Checked 2026-11-03 against the repository instruction base.
| Claim | Source | Checked |
|---|---|---|
| Case shape, grader hierarchy, judge calibration, and reporting rules | `instructions/validation/references/evaluation-design.md` | 2026-11-03 |
| Aggregation, `pass^k` versus `pass@k`, the record fields, and error-trial accounting | `instructions/harness-engineering/HARNESS_ENGINEERING.md`, `instructions/harness-engineering/references/measurement.md` | 2026-11-03 |
| Trigger case composition and per-case run counts | `instructions/skill/references/trigger-design.md` | 2026-11-03 |
No external URL is cited in this file; every rule is a restatement of the repository base above.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization. | Stars: 469k | Updated | Claude CodeCodex | Verified | Install |
| academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file). | Stars: 197k | Updated | Claude CodeCodex | Unverified | Install |
| doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |