Disentinel/grafema / .claude/skills/swe-bench-grafema-experiments
swe-bench-grafema-experiments
VerifiedSWE-bench pipeline for A/B testing Grafema with Claude Code. Use when: (1) running SWE-bench experiments, (2) comparing baseline vs grafema conditions, (3) evaluating results, (4) debugging container issues.
SWE-bench Grafema Experiments
Pipeline Overview
Uses claude -p inside SWE-bench Docker containers. Two conditions:
- Baseline: Claude Code + standard tools
- Grafema: Claude Code +
grafemainstalled with pre-built graph + MCP tools
Same agent, same environment, same prompt (except tool docs section).
Quick Commands
# Generate tasks.json (one-time)
python scripts/swe-bench/generate-tasks.py > scripts/swe-bench/tasks.json
# Run single task
./scripts/swe-bench/run.sh axios__axios-4731 --mode baseline
./scripts/swe-bench/run.sh axios__axios-4731 --mode grafema
# Run all JS/TS tasks (loop)
for task in $(jq -r '.[].instance_id' scripts/swe-bench/tasks.json); do
./scripts/swe-bench/run.sh "$task" --mode baseline
./scripts/swe-bench/run.sh "$task" --mode grafema
done
# Compare results
./scripts/swe-bench/compare.sh
# Evaluate with swebench
source /Users/vadimr/swe-bench-research/mini-swe-agent/.venv/bin/activate
python -m swebench.harness.run_evaluation \
--dataset_name swe-bench/SWE-Bench_Multilingual \
--predictions_path scripts/swe-bench/results/baseline/preds.jsonl \
--max_workers 1 --run_id baseline
Key Files
| File | Purpose |
|---|---|
scripts/swe-bench/run.sh | Main pipeline script |
scripts/swe-bench/compare.sh | Results comparison |
scripts/swe-bench/generate-tasks.py | Task generation from HuggingFace |
scripts/swe-bench/tasks.json | Pre-cached 43 JS/TS tasks |
scripts/swe-bench/templates/prompt-baseline.md | Baseline prompt |
scripts/swe-bench/templates/prompt-grafema.md | Grafema prompt (with tool docs) |
_ai/swe-bench-runbook.md | Full runbook |
Debugging
Container issues
# Check image exists
docker images | grep sweb.eval
# Check Node version
docker run --rm <image> node --version
# Debug inside container
docker run -it --name debug-swe -v ~/.claude:/root/.claude:ro <image> bash
Auth issues
# Verify host auth
claude --version
# Check mount inside container
docker exec <container> ls -la /root/.claude/
Grafema issues
# Check if grafema installed
docker exec <container> grafema --version
# Check if graph was built
docker exec <container> ls /testbed/.grafema/
# Check MCP config
docker exec <container> cat /testbed/.mcp.json
Historical Results
Grafema consistently reduces file exploration (39-100%) but hasn't improved fix correctness in tested tasks. The new claude -p pipeline should provide cleaner measurements.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |