agentsope/SkillAlchemy / skills/agentsop-trajectory-to-semantic-ops
agentsop-trajectory-to-semantic-ops
VerifiedSOP for distilling reusable semantic operators (data-science operation skills) from agent execution trajectories: read action/observation traces, classify per-task failures, bind data-semantic constraints (input semantics / operation decision / output constraints / validation), and feed bounded edits back through an optimizer such as SkillOpt. Use when mining agent runs for cross-task operation knowledge, when a data-science skill keeps failing on the same operation family (join cardinality, group granularity, unit, time alignment), or when wiring trajectory history into a skill optimizer without polluting the skill body.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.7 KB |
| SKILL.md | skill md | 11.6 KB |
| intermediate/operation_candidates.json | config | 8.6 KB |
| references/R1-architecture.md | markdown | 5.8 KB |
| references/R2-sop-workflow.md | markdown | 4.5 KB |
| references/R3-dilemma-cases.md | markdown | 3.8 KB |
| references/R4-anti-patterns.md | markdown | 3.1 KB |
| references/R5-ecosystem-context.md | markdown | 4.1 KB |
references/R4-anti-patterns.md
View on GitHub# R4: Anti-Patterns, Mistakes & Hard Boundaries
## When NOT to do trajectory → semantic-ops extraction
| Situation | Alternative |
|---|---|
| You only have a single run, no held-out set | You can't separate mechanism from noise; collect more before extracting |
| The error is an environment artifact (PowerShell quoting, `D:\` path in Linux container) | Fix the harness, not the skill — a skill edit can't fix a broken executor |
| The task is one-off and never recurs | A pasted answer "works" but a reusable operator is wasted effort |
| You want to *prove* "progressive disclosure helps" without a flat-load baseline | You're testing organization, not knowledge; need same-content structure control |
## Common mistakes
1. **Writing instance answers / task IDs / test-set column names into the skill.** Overfits, pollutes the operator, leaks evaluation. Constraint survives task change; an answer does not.
2. **Injecting history material and forgetting to strip it.** The main file balloons (9873 chars) and blows the executor timeout. Always `strip_history_text()`.
3. **Gating on 4 single-run tasks.** ±0.22 noise ≈ the signal you're trying to detect. Accept/reject becomes a coin flip.
4. **Reversing the previous generation's rule.** Rules fighting across generations is a direct cause of oscillation. Forbid reversals, cap diff size.
5. **Treating "invocation count" as "adoption".** Count discovery / activation / load / adoption / outcome separately.
6. **Claiming "verified" where you only did a type/row-count check.** Fuzzy-match semantic correctness is only partially caught by such checks.
7. **Using single-run training scores to steer optimization.** ±0.125 per-task noise drowns the signal; use repeat ≥ 3.
8. **Reading `exit_code=0` as "correct".** The agent can return 0 while computing the wrong granularity, unit, or window.
## Hard boundaries
- **Produce constraints, not answers.** The operator's value is a cross-task decision rule; a pasted pipeline implementation is not knowledge.
- **Oracle routing is diagnosis, not a result.** Using gold skills / gold code / gold dependencies to route is an oracle diagnostic — never fold it into the main reported score.
- **Don't preset "deeper disclosure is better".** Progressive-disclosure benefit depends on scale and harness (Skill Blocks, SkillJuror). It's a hypothesis to test, not an axiom.
- **Correlation ≠ mechanism.** Score moving with invocation count proves nothing about *why*. Isolate content, organization, and invocation.
- **Freeze the environment before comparing.** Model, harness, image, dependencies, task interface, plotting protocol, timeout, retry — fix them, then vary only the skill. Otherwise the gain is unattributable.
- **Honor the train/gate/test split.** Never reuse the same tasks across them; group by data-source family to avoid leakage, not by random task split.
## References
- `AgenticDataBench/docs/research/IDEA_ASSESSMENT_2026-09-19.md` (§四–§九)
- `AgenticDataBench/testbed/SkillOpt_两种优化方式分析.md` (bloat, gate noise, reversal)
- `AgenticDataBench/testbed/SKILLOPT_HISTORY_PATCH.md` (bundle vs flat, strip round-trip)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |