Skip to content

sb-dev/openquant / .agents/skills/spec-to-code-compliance

spec-to-code-compliance

Verified

Check implementation code against its specification, protocol, whitepaper, or design document. Trace each requirement to evidence, report contradictions and missing or partial enforcement, and find undocumented behavior.

FileKindSize
SKILL.mdskill md7.0 KB
upstream/.claude-plugin/plugin.jsonconfig370 B
upstream/README.mdmarkdown4.1 KB
upstream/agents/spec-compliance-checker.mdmarkdown5.7 KB
upstream/evals/.gitignoreother9 B
upstream/evals/documents-contradict/case.yamlconfig2.0 KB
upstream/evals/documents-contradict/fixture/README.mdmarkdown821 B
upstream/evals/documents-contradict/fixture/SPEC.mdmarkdown1.5 KB
upstream/evals/documents-contradict/fixture/Vault.solother2.8 KB
upstream/evals/documents-contradict/graders/reports-the-contradiction.mdmarkdown1.4 KB
upstream/evals/name-is-not-evidence/case.yamlconfig2.2 KB
upstream/evals/name-is-not-evidence/fixture/SPEC.mdmarkdown1.5 KB
upstream/evals/name-is-not-evidence/fixture/Vault.solother2.8 KB
upstream/evals/name-is-not-evidence/graders/senior-branch-unenforced.mdmarkdown1.5 KB
upstream/evals/name-is-not-evidence/graders/verdict-per-requirement.mdmarkdown1.0 KB
upstream/evals/routes-not-inline/case.yamlconfig2.9 KB
upstream/evals/routes-not-inline/fixture/SPEC.mdmarkdown1.5 KB
upstream/evals/routes-not-inline/fixture/Vault.solother2.8 KB
upstream/evals/routes-not-inline/graders/dispatches-the-workflow.mdmarkdown109 B
upstream/evals/routes-not-inline/graders/invokes-the-skill.mdmarkdown90 B
upstream/skills/spec-to-code-compliance/SKILL.upstream.mdmarkdown7.4 KB
upstream/skills/spec-to-code-compliance/agents/openai.yamlconfig252 B
upstream/skills/spec-to-code-compliance/assets/trail-of-bits-mark.svgother3.0 KB
upstream/skills/spec-to-code-compliance/resources/ANALYSIS_FORMAT.mdmarkdown2.8 KB
upstream/skills/spec-to-code-compliance/resources/DIVERGENCE_RUBRIC.mdmarkdown3.3 KB
upstream/skills/spec-to-code-compliance/resources/DOMAIN_NOTES.mdmarkdown5.7 KB
upstream/skills/spec-to-code-compliance/resources/WORKED_EXAMPLE.mdmarkdown4.1 KB
upstream/workflows/spec-compliance.jsscript28.4 KB

upstream/evals/name-is-not-evidence/case.yaml

View on GitHub
schema_version: "1.1"
name: name-is-not-evidence
description: >
  The plugin's central judgment rule is that a requirement is not satisfied because a plausibly-named function
  guards it — the enforcement has to be opened and read. Here §3.2 says a redemption must not push a balance
  below locked collateral "regardless of tier", and redeem() guards itself with
  require(_collateralPreserved(...)), which returns true unconditionally for Senior members without ever
  comparing balance to locked. Reaching the right verdict requires opening the callee and walking both
  branches; stopping at the call site produces a confident "implemented" that is wrong. The fixture also
  carries an easier divergence (the operator can zero a balance via reassign, which §4 forbids) so that a
  shallow pass still produces findings and the grader is not measuring whether the model found anything at all.
  Measured Δ 0.00: 1.00 with the plugin, 1.00 without it, both in 4 turns. Opus 5 opens the callee and finds
  the Senior branch unaided, so this case currently scores the model rather than the skill. It is kept as a
  regression guard — it will go red if a future edit stops the skill reading callees — but it is not evidence
  that the plugin adds anything, and it should not be cited as though it were. Making it discriminate means
  burying the branch further from the entry point, or giving the fixture enough surface that a single pass has
  to choose where to look.
tags: [callee-reading, verdict-discipline, no-delta]

context:
  add_dirs: [fixture]

execution:
  prompt: |
    SPEC.md in the fixture directory is the protocol specification for Vault.sol, which is in the same
    directory.

    Check the contract against section 3 of the spec. I want to know which of those requirements the code
    actually holds to.
  max_turns: 30
  timeout_seconds: 900
  allowed_tools: [Read, Glob, Grep, Write, Skill, Task]

runs: 3

expected_outcome: >
  The response reports that §3.2 is not enforced for Senior-tier members, because _collateralPreserved returns
  true on the Senior branch without comparing balance to locked collateral, and does not report §3.2 as
  satisfied on the strength of the require at the call site.
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall