cskwork/supergoal-skill / skills/supergoal
supergoal
UnverifiedRoute objective-driven development, verification, planning, "teach/explain", codebase learning, and skill evaluation through the Supergoal workflow. Use when Supergoal is requested or an autonomous delivery workflow is needed.
templates/harness-eval-external/deepswe/task-set.yaml
View on GitHubid: external-deepswe-v1.1-public-pilot
title: "DeepSWE v1.1 public full-cycle task set"
case_type: external_benchmark
benchmark:
name: DeepSWE
site_url: "https://deepswe.datacurve.ai/"
repo_url: "https://github.com/datacurve-ai/deep-swe"
benchmark_ref: "3cda4081fed96103a6395de39c85e9b20275e307"
task_count_at_ref: 113
verifier_outputs:
- "verifier/reward.json"
- "verifier/ctrf.json"
- "verifier/test-stdout.txt"
- "verifier/run.log"
default_task_id: etree-xml-diff-patch
default_effectiveness_task_id: etree-xml-diff-patch
default_forced_suite_id: forced-default-deepswe-difficult-swe
default_forced_suite_task_ids:
- etree-xml-diff-patch
- cliffy-config-file-parsing
- csstree-shorthand-expansion-compression
- skrub-duration-encoding
- termenv-preserve-ansi-resets
smoke_task_id: happy-dom-abort-pending-body-reads
difficulty_evidence:
source: "DeepSWE public leaderboard trial records scraped per task page, snapshot 2026-07-06"
method: "13108 included-in-score trials across 21 models and 113 tasks; per-task overall pass rate plus gpt-5.5 (mini-swe-agent, medium/high/xhigh) pass count. Suite tasks need low overall pass rate (difficult) AND nonzero gpt-5.5 passes (not a guaranteed floor for the codex gpt-5.5 lane); etree stays as the locally proven-headroom anchor."
suite_rule: "Prefer overall pass <= 20 percent with gpt-5.5 between 1/12 and 9/12; spread languages; avoid rust tasks in the 900s low-effort lane (compile budget risk)."
held_out_escalation_pool:
- updo-policy-alerting
- happy-dom-deterministic-intersectionobserver
- testem-bail-on-test-failure
- koota-deferred-mutation-buffer
- dateutil-rfc5545-timezone-interop
stop_policy:
timeout_seconds: 900
valid_outcomes:
- completed
- budget_timeout
- error
manual_interrupt: invalid_paired_correctness
capture_patch_on_budget_timeout: true
score_patch_after_terminate_timeout_or_error: true
adapter_policy:
preferred_official: "DeepSWE/Pier committed-code grading in a clean isolated environment."
codex_readonly_git_fallback: "If the Codex workspace sandbox makes .git read-only, capture working-tree diffs for both arms after terminate/timeout/error and score those patches with the same verifier."
eval_intent:
goal: "Measure whether supergoal improves public, complex source-code task success versus the same agent without supergoal."
constraints:
- "Use internet-available open-source tasks, not private LMS code."
- "Use the same benchmark task, model, agent, repo snapshot, and verifier for both arms."
- "Keep hidden verifier files, solution patches, grader code, and tests out of the harness prompt."
tradeoffs:
- "A harness prompt may cost more tokens; claim a win only when pass-rate or partial reward improves enough to justify the cost."
rejected_approaches:
- "Treating the local authz-cache u3 fixture as a harness win; it only proves hidden-test discrimination."
- "Using private LMS code that cannot be committed or independently replayed."
- "Treating the 2026-07-03 Happy DOM interrupted harness arm as a valid correctness result."
- "Treating a completed Happy DOM tie with perfect baseline score as meaningful harness lift."
- "Keeping yjs-map-conflict-detection in the difficult suite after leaderboard data (rank 101/113) and a local perfect-baseline run both showed no headroom."
- "Selecting suite tasks by category label instead of measured pass rates."
- "Rust tasks in the 900s low-effort lane; compile time eats the agent budget."
arms:
baseline:
prompt_condition: "Original DeepSWE instruction only."
harness:
prompt_condition: "Original DeepSWE instruction plus generated supergoal harness prefix from templates/harness-eval-external/deepswe/prepare-supergoal-arm.mjs."
role_source: "shipped_files"
required_source_files:
- "SKILL.md"
- "reference/role-loop.md"
- "reference/delivery-gate.md"
- "agents/qa-auditor.md"
metrics:
correctness:
primary: "task pass rate from reward.json binary reward."
secondary: "pass_fraction or failed-test count from reward.json/ctrf.json when available."
lift: "absolute percentage-point delta; relative percent lift only as secondary context."
cost:
primary: "total_tokens per run from agent/Pier trajectory metadata."
speed:
primary: "duration_ms or wall-clock seconds per run from Pier metadata."
routing_accuracy:
primary: "not applicable unless this run changes the supergoal trigger/router; otherwise reuse the 20-prompt routing probe."
decision_rule:
forced_default: "For difficult SWE or harness-effectiveness claims without a named task, run the five-task forced DeepSWE suite: etree-xml-diff-patch, cliffy-config-file-parsing, csstree-shorthand-expansion-compression, skrub-duration-encoding, and termenv-preserve-ansi-resets."
directional_pilot: "At least 3 different paired tasks or 3 paired seeds per task; report per-task/per-seed vectors. One run per task is directional only."
proven_claim: "At least 6 paired seeds per arm across multiple tasks, paired McNemar p < 0.05 on binary task outcomes, and no unacceptable cost/time regression."
no_signal: "If both arms pass every selected task, report not_proven_no_headroom and select harder held-out tasks."
tasks:
- id: etree-xml-diff-patch
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/etree-xml-diff-patch"
local_path: "tasks/etree-xml-diff-patch"
upstream_repo: "https://github.com/beevik/etree"
base_commit: "4032e04c8f2e2f35e43ce5d772fcef14a5df4d74"
category: feature_request
language: go
why_selected: "Default scoring task: XML diff, patch, reverse patch, three-way merge, and summaries require larger domain implementation breadth than the saturated Happy DOM smoke task."
public_headroom_note: "Selected after Happy DOM saturated current Codex gpt-5.5 low reasoning. Do not call it proven-discriminating until a completed paired run shows baseline headroom or a nonzero harness-vs-baseline delta."
effectiveness_default: true
minimum_pilot: true
- id: happy-dom-abort-pending-body-reads
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/happy-dom-abort-pending-body-reads"
local_path: "tasks/happy-dom-abort-pending-body-reads"
upstream_repo: "https://github.com/capricorn86/happy-dom"
base_commit: "82a0888cb2c87a6123e05424b528f8e8c9b3e426"
category: bugfix
language: typescript
why_selected: "Smoke/reliability task: async shutdown bugfix with pending body reads, formData parsing, timers, navigation cleanup, and preservation checks."
public_headroom_note: "The interrupted 2026-07-03 setup run showed only diagnostic headroom. The later no-interrupt Codex gpt-5.5 low run saturated both arms: baseline and harness reward=1, f2p=14/14, p2p=165/165. Keep this as a public smoke/full-cycle reliability task only."
smoke_only: true
minimum_pilot: false
- id: cliffy-config-file-parsing
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/cliffy-config-file-parsing"
local_path: "tasks/cliffy-config-file-parsing"
upstream_repo: "https://github.com/c4spar/cliffy"
base_commit: "132a437c40cffbdfbe474ca808c8debde59e2633"
category: feature_request
language: typescript
why_selected: "Secondary broad feature task with parser, precedence, inheritance, caching, and validation edges."
public_headroom_note: "Previously too broad for the low-turn Claude pilot: both arms exceeded budget and produced no patch."
minimum_pilot: false
- id: yjs-map-conflict-detection
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/yjs-map-conflict-detection"
local_path: "tasks/yjs-map-conflict-detection"
upstream_repo: "https://github.com/yjs/yjs"
base_commit: "7795050a749bd1111cbbdd9d0219b27226a8e710"
category: feature_request
language: javascript
why_selected: "CRDT-style map conflict detection requires domain reasoning about deterministic concurrent writes."
public_headroom_note: "Demoted from the forced suite 2026-07-06: leaderboard rank 101/113 (55.2 percent overall, gpt-5.5 11/12), and the local Spark-low paired run had a perfect baseline that the harness regressed by one preservation check. Too easy for a difficult-SWE suite; no baseline headroom."
suite_default: false
minimum_pilot: false
- id: csstree-shorthand-expansion-compression
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/csstree-shorthand-expansion-compression"
local_path: "tasks/csstree-shorthand-expansion-compression"
upstream_repo: "https://github.com/csstree/csstree"
base_commit: "88e3d965c0b1628642a30a841745b410d6835052"
category: feature_request
language: javascript
why_selected: "Shorthand expansion/compression in a CSS lexer: many property grammars and edge interactions. Leaderboard rank 15/113 (11.2 percent overall); gpt-5.5 6/12 (high 2/4, xhigh 4/4) so the codex gpt-5.5 lane has headroom without a guaranteed floor."
suite_default: true
minimum_pilot: false
- id: skrub-duration-encoding
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/skrub-duration-encoding"
local_path: "tasks/skrub-duration-encoding"
upstream_repo: "https://github.com/skrub-data/skrub"
base_commit: "24c4466fea94f551fb73d21eba54038dc5b346d3"
category: feature_request
language: python
why_selected: "Duration encoding inside TableVectorizer: dataframe-library edge behavior plus API integration. Leaderboard rank 14/113 (10.3 percent overall); gpt-5.5 4/12 (all at xhigh) - hard but solvable by the model family, and it adds python coverage."
suite_default: true
minimum_pilot: false
- id: termenv-preserve-ansi-resets
task_url: "https://deepswe.datacurve.ai/data/v1/tasks/termenv-preserve-ansi-resets"
local_path: "tasks/termenv-preserve-ansi-resets"
upstream_repo: "https://github.com/muesli/termenv"
base_commit: "368a3572b8146cc038b3f240da6792003d7e42c5"
category: feature_request
language: go
why_selected: "ANSI reset preservation during truncation/styling: stateful escape-sequence tracking with subtle correctness edges. Leaderboard rank 11/113 (9.5 percent overall); gpt-5.5 5/12 spread across efforts. Small repo, fast test cycle - fits the 900s low-effort budget."
suite_default: true
minimum_pilot: false
commands:
default_forced_suite_runner:
- "node templates/harness-eval-external/deepswe/run-default-suite.mjs --agent codex --model gpt-5.5 --reasoning-effort low --codex-auth-json auto --timeout-seconds 900 --run-root /tmp/sg-deepswe-default-suite"
full_cycle_runner:
- "node templates/harness-eval-external/deepswe/run-full-cycle.mjs --task etree-xml-diff-patch --agent codex --model gpt-5.5 --reasoning-effort low --codex-auth-json auto --timeout-seconds 900 --run-root /tmp/sg-deepswe-etree-full-cycle"
smoke_full_cycle_runner:
- "node templates/harness-eval-external/deepswe/run-full-cycle.mjs --task happy-dom-abort-pending-body-reads --agent codex --model gpt-5.5 --reasoning-effort low --codex-auth-json auto --timeout-seconds 900 --run-root /tmp/sg-deepswe-happy-dom-full-cycle"
full_cycle_dry_run:
- "node templates/harness-eval-external/deepswe/run-full-cycle.mjs --dry-run --task etree-xml-diff-patch --agent codex --model gpt-5.5 --reasoning-effort low --codex-auth-json auto --timeout-seconds 900 --run-root /tmp/sg-deepswe-etree-full-cycle"
clone_benchmark:
- "git clone https://github.com/datacurve-ai/deep-swe /tmp/deep-swe"
- "git -C /tmp/deep-swe checkout 3cda4081fed96103a6395de39c85e9b20275e307"
install_runner:
- "uv tool install datacurve-pier"
prepare_harness_task:
- "node templates/harness-eval-external/deepswe/prepare-supergoal-arm.mjs /tmp/deep-swe /tmp/deep-swe-supergoal etree-xml-diff-patch ."
baseline_example:
- "pier run -p /tmp/deep-swe/tasks/etree-xml-diff-patch --agent mini-swe-agent --model openai/gpt-5.5"
harness_example:
- "pier run -p /tmp/deep-swe-supergoal/tasks/etree-xml-diff-patch --agent mini-swe-agent --model openai/gpt-5.5"
artifacts:
required_per_run:
- "Pier trajectory metadata"
- "verifier/reward.json"
- "verifier/ctrf.json"
- "verifier/test-stdout.txt"
- "verifier/run.log"
- "model.patch"
- "harness-metadata.json for harness arm"
- "stop policy and actual process outcome"
- "full-cycle runner manifest.json, summary.json, report.md, and per-arm logs"
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |