Skip to content

Disentinel/grafema / .claude/skills/swe-bench-grafema-experiments

swe-bench-grafema-experiments

Verified

SWE-bench pipeline for A/B testing Grafema with Claude Code. Use when: (1) running SWE-bench experiments, (2) comparing baseline vs grafema conditions, (3) evaluating results, (4) debugging container issues.

SWE-bench Grafema Experiments

Pipeline Overview

Uses claude -p inside SWE-bench Docker containers. Two conditions:

  • Baseline: Claude Code + standard tools
  • Grafema: Claude Code + grafema installed with pre-built graph + MCP tools

Same agent, same environment, same prompt (except tool docs section).

Quick Commands

# Generate tasks.json (one-time)
python scripts/swe-bench/generate-tasks.py > scripts/swe-bench/tasks.json

# Run single task
./scripts/swe-bench/run.sh axios__axios-4731 --mode baseline
./scripts/swe-bench/run.sh axios__axios-4731 --mode grafema

# Run all JS/TS tasks (loop)
for task in $(jq -r '.[].instance_id' scripts/swe-bench/tasks.json); do
  ./scripts/swe-bench/run.sh "$task" --mode baseline
  ./scripts/swe-bench/run.sh "$task" --mode grafema
done

# Compare results
./scripts/swe-bench/compare.sh

# Evaluate with swebench
source /Users/vadimr/swe-bench-research/mini-swe-agent/.venv/bin/activate
python -m swebench.harness.run_evaluation \
  --dataset_name swe-bench/SWE-Bench_Multilingual \
  --predictions_path scripts/swe-bench/results/baseline/preds.jsonl \
  --max_workers 1 --run_id baseline

Key Files

FilePurpose
scripts/swe-bench/run.shMain pipeline script
scripts/swe-bench/compare.shResults comparison
scripts/swe-bench/generate-tasks.pyTask generation from HuggingFace
scripts/swe-bench/tasks.jsonPre-cached 43 JS/TS tasks
scripts/swe-bench/templates/prompt-baseline.mdBaseline prompt
scripts/swe-bench/templates/prompt-grafema.mdGrafema prompt (with tool docs)
_ai/swe-bench-runbook.mdFull runbook

Debugging

Container issues

# Check image exists
docker images | grep sweb.eval

# Check Node version
docker run --rm <image> node --version

# Debug inside container
docker run -it --name debug-swe -v ~/.claude:/root/.claude:ro <image> bash

Auth issues

# Verify host auth
claude --version

# Check mount inside container
docker exec <container> ls -la /root/.claude/

Grafema issues

# Check if grafema installed
docker exec <container> grafema --version

# Check if graph was built
docker exec <container> ls /testbed/.grafema/

# Check MCP config
docker exec <container> cat /testbed/.mcp.json

Historical Results

Grafema consistently reduces file exploration (39-100%) but hasn't improved fix correctness in tested tasks. The new claude -p pipeline should provide cleaner measurements.

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall