lidge-jun/codexclaw / plugins/codexclaw/skills/dev-testing
cxc-dev-testing
VerifiedUse for tests and release verification. Triggers: regression test, Playwright, E2E, contract test, coverage, CI flake, TDD, 테스트, 회귀 테스트, 품질 게이트.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.2 KB |
| agents/openai.yaml | config | 164 B |
| examples/console_logging.py | script | 1.1 KB |
| examples/element_discovery.py | script | 1.5 KB |
| examples/static_html_automation.py | script | 1.1 KB |
| references/acceptance-gates.md | markdown | 3.4 KB |
| references/acceptance-row-reachability.md | markdown | 1.1 KB |
| references/api-test-patterns.md | markdown | 3.4 KB |
| references/backend-testing.md | markdown | 4.3 KB |
| references/browser-testing.md | markdown | 3.5 KB |
| references/ci-integration.md | markdown | 1.3 KB |
| references/ci-pipeline.md | markdown | 8.5 KB |
| references/core/crud-test-matrix.md | markdown | 2.2 KB |
| references/edge-first-testing.md | markdown | 2.5 KB |
| references/limited-oracle-evaluation.md | markdown | 2.7 KB |
| references/load-testing.md | markdown | 6.9 KB |
| references/ml-evaluation.md | markdown | 7.2 KB |
| references/patch-integrity.md | markdown | 1.4 KB |
| references/property-and-mutation-testing.md | markdown | 652 B |
| references/tdd-and-regressions.md | markdown | 5.4 KB |
| references/test-oracle-integrity.md | markdown | 2.0 KB |
| references/test-strategy.md | markdown | 1.5 KB |
| scripts/with_server.py | script | 3.6 KB |
references/limited-oracle-evaluation.md
View on GitHub# Limited-Oracle / Score-Objective Evaluation
Use this reference when the real evaluator is scarce, paid, rate-limited, or opaque, and
local tests are proxy metrics for a score/objective. These rules are paired with
`cxc-pabcd` Optimization-Loop Meta-Rules (plateau discipline). They were observed in a
14-discard optimization plateau where a prefix-only replay gate and hard
draw-protection invariant locked a 3.5/8 score.
- **GATE-ORACLE-VALIDITY-01 (STRICT):** When the true evaluator/oracle is
rate-limited and local metrics are proxies, evaluator validity is a PREREQUISITE
gate. Before trusting the proxy for accept/reject, quantify historical divergence:
cases where the proxy said better/equal but the oracle said worse. A proxy with known
optimistic bias must not be the sole acceptance evidence.
- **GATE-PREFIX-HORIZON-01 (DEFAULT):** Replay-based evidence, such as recorded logs or
scripted opponents, is prefix-valid only. It stops being valid as soon as the
candidate diverges from the recorded trajectory. Candidates that diverge early need
live adversarial evaluation with a modeled opponent/environment, not replay-only
acceptance. State the divergence turn/point whenever citing replay evidence.
- **GATE-INVARIANT-EV-01 (DEFAULT):** Every hard invariant in an acceptance gate, meaning
a metric that must not regress, needs an expected-value justification: the value it
protects versus the candidate-space it vetoes. If a hard invariant vetoes three or
more consecutive candidates that target strictly larger gains, downgrade it to a
soft cost and re-justify or remove it.
- **GATE-HOLDOUT-LEAKAGE-01 (DEFAULT):** Fixed evaluation assets - recorded logs, test
maps, sparring bots, graders, and even public oracle outcomes - become training data
once candidates are repeatedly tuned against them (adaptive reuse overfits the
holdout itself; Blum & Hardt's Ladder, arXiv:1502.04585, and the reusable-holdout
line, arXiv:1506.02629). Rotate or expand the gate's instance set as tuning
accumulates, and reserve a blind slice (instances never used for candidate
selection) as the final acceptance check. A candidate that wins only on the tuned
set and not on the blind slice is gate-overfit, not improved.
- **GATE-AGREEMENT-STATS-01 (HEURISTIC):** Calibrate a proxy against the scarce oracle
with agreement statistics, not correlation alone: sign-discordance rate (proxy said
better/equal, oracle said worse), mean and worst-case proxy-minus-oracle error, and
rank agreement on paired decisions (Bland-Altman method-comparison doctrine). Track
these per scenario family - a proxy can be trustworthy on one instance class and
optimistic on another - and re-derive them whenever the oracle returns new results.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |