agentsope/SkillAlchemy / skills/agentsop-code-execution-decision
agentsop-code-execution-decision
VerifiedDecision rubric for when an LM agent should write-and-run code (Program-of-Thought / code interpreter) versus reason in natural language: classify each step as deterministic- computable (emit + execute code, feed the result back) vs judgment (stay in prose). Use when designing or debugging an agent step that does arithmetic/parsing/data transforms, when prose reasoning hallucinates a computation (under-coding), or when a sandbox round- trip is wasted on a judgment task (over-coding). Search keywords: code interpreter, agent does math wrong, calculator hallucination, when to run code vs reason, program of thought, PoT, tool vs reasoning.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 1.9 KB |
| SKILL.md | skill md | 25.5 KB |
| intermediate/operation_candidates.json | config | 4.2 KB |
| references/R1-source-evidence.md | markdown | 4.6 KB |
intermediate/operation_candidates.json
View on GitHub{
"skill": "code-execution-decision",
"version": "0.1.0",
"type": "enhancement-overlay",
"overlays": "dspy (ProgramOfThought mechanism; missing decision rubric)",
"core_thesis": "LMs are unreliable calculators but reliable coders. Emit code when the answer needs determinism/precision; reason in prose when it needs judgment.",
"failure_asymmetry": {
"under_coding": "reasoning through a computation -> silent plausible-wrong number",
"over_coding": "sandboxing a judgment task -> loud, cheap wasted round-trip",
"default_lean": "when genuinely uncertain AND a verifiable core exists, lean toward code"
},
"operation_candidates": [
{
"id": "op1",
"name": "computable-vs-judgment gate",
"trigger": "any step about to be reasoned",
"action": "ask: single verifiable answer a program could check? yes->code, no->prose, mixed->decompose",
"output": "routing decision",
"evidence": "PoT premise arXiv 2211.12588; DSPy ChainOfThought vs ProgramOfThought",
"kept": true
},
{
"id": "op2",
"name": "decompose mixed steps",
"trigger": "step contains both a number and a narrative",
"action": "split into computable sub-step (code) + judgment sub-step (prose)",
"output": "two routed sub-steps",
"evidence": "mirrors output-format-by-model mixed-content split (Case B)",
"kept": true
},
{
"id": "op3",
"name": "sandbox choice",
"trigger": "decided to emit code",
"action": "pick interpreter by trust + capability (DSPy PoT / OpenAI Code Interpreter / Anthropic code-exec VM / LangChain PythonREPLTool=unsandboxed)",
"output": "chosen runtime",
"evidence": "DSPy modules docs; LangChain PythonREPLTool docs; cross-link output-format-by-model for wire shape",
"kept": true
},
{
"id": "op4",
"name": "result-back-into-LM",
"trigger": "code produced a value",
"action": "inject stdout/return value into next LM turn so model narrates/uses it",
"output": "LM-consumed result",
"evidence": "PoT design: code computes, LM contextualizes",
"kept": true
},
{
"id": "op5",
"name": "error retry (bounded)",
"trigger": "code raised a traceback",
"action": "feed error to LM, regenerate, re-run; cap at max_iters (~3) then fall back",
"output": "fixed code or graceful fallback",
"evidence": "DSPy ProgramOfThought max_iters; cross-link test-fix-loop",
"kept": true
},
{
"id": "op6",
"name": "precision escalation",
"trigger": "prose answer involves multi-step arithmetic",
"action": "re-route the arithmetic to code even if prose started it",
"output": "code-verified number",
"evidence": "LMs unreliable calculators; PAL arXiv 2211.10435",
"kept": true
},
{
"id": "op7",
"name": "over-coding veto",
"trigger": "about to sandbox a judgment task",
"action": "stop: no deterministic core -> reasoning not code",
"output": "prose reasoning, no sandbox call",
"evidence": "Case B; avoids wasted round-trip",
"kept": true
},
{
"id": "op8-discarded",
"name": "always-code-then-verify",
"trigger": "any answer",
"action": "emit code for every step then verify",
"output": "n/a",
"evidence": "none",
"kept": false,
"reason_discarded": "this IS the over-coding anti-pattern; folded into op7 veto instead of being its own op"
}
],
"dilemma_cases": [
{
"id": "A",
"title": "arithmetic in prose hallucinates -> PoT fixes (under-coding)",
"extractable_op": "op6 - multi-step arithmetic in prose is a smell; re-route to code"
},
{
"id": "B",
"title": "over-coding a judgment task wastes a sandbox round-trip",
"extractable_op": "op7 - no verifiable answer -> no code; veto the round-trip"
},
{
"id": "C",
"title": "mixed step (summary + total) -> decompose",
"extractable_op": "op2 - one task != one strategy; gate per step"
}
],
"cross_links": ["output-format-by-model", "test-fix-loop", "dspy"],
"frameworks_compared": [
"DSPy ProgramOfThought",
"OpenAI Code Interpreter",
"Anthropic code-execution tool",
"LangChain PythonREPLTool"
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation). | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |