agentsope/SkillAlchemy / skills/agentsop-trajectory-to-semantic-ops
agentsop-trajectory-to-semantic-ops
VerifiedSOP for distilling reusable semantic operators (data-science operation skills) from agent execution trajectories: read action/observation traces, classify per-task failures, bind data-semantic constraints (input semantics / operation decision / output constraints / validation), and feed bounded edits back through an optimizer such as SkillOpt. Use when mining agent runs for cross-task operation knowledge, when a data-science skill keeps failing on the same operation family (join cardinality, group granularity, unit, time alignment), or when wiring trajectory history into a skill optimizer without polluting the skill body.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.7 KB |
| SKILL.md | skill md | 11.6 KB |
| intermediate/operation_candidates.json | config | 8.6 KB |
| references/R1-architecture.md | markdown | 5.8 KB |
| references/R2-sop-workflow.md | markdown | 4.5 KB |
| references/R3-dilemma-cases.md | markdown | 3.8 KB |
| references/R4-anti-patterns.md | markdown | 3.1 KB |
| references/R5-ecosystem-context.md | markdown | 4.1 KB |
references/R1-architecture.md
View on GitHub# R1: Trajectory → Semantic-Ops Architecture & Mental Model
## Core insight
The whole loop is **"collect the evidence, then translate it into constraints."** You never hand the raw trajectory back to the model; you first turn it into a *diagnosis* (what failed, and which layer it failed at), and only the diagnosis becomes a reusable operator field.
```
+-----------------------------------------------------------------------+
| execution trajectory (kind: message | command, exit_code, output) |
| | |
| v |
| compress (动作 → 观察, 优先 exit≠0 步) |
| | |
| v |
| classify (persistent_fail / regressed / flat_partial / improved / |
| stable_success) |
| | |
| v |
| synthesize (分数 + 趋势 + 失败轨迹 → markdown 材料) |
| | |
| v |
| inject → optimize (有界编辑 + 门控) → strip (剥除材料) |
| | |
| v |
| semantic operator (适用条件 / 输入语义 / 操作决策 / 输出约束 / 验证) |
+-----------------------------------------------------------------------+
```
## The trajectory schema (grounded, not invented)
An execution trajectory is a JSON array of step objects. Two kinds appear:
```json
{"kind":"message","content":"... agent narration ..."}
{"kind":"command","command":"bash -lc 'python3 - <<PY ...'","exit_code":0,"output":"... observation ..."}
```
The same content is also persisted as human-readable `agent_trace.txt` (rendered as `[command] … exit=N 观察: …`) and scored in `results.jsonl` (fields `id` / `hard` / `soft` / `fail_reason` / `workspace`, plus timing/token metrics in later iterations).
Three reading rules:
| Rule | Why it matters |
|---|---|
| `exit_code` is where it *broke*, not where it *erred* | A step can return 0 yet compute the wrong granularity; a step can return 1 from a quote-escaping bug, not a data bug |
| `message` is self-narration, not ground truth | The agent says "已完成" while `output.csv` is missing — trust command/output/score over prose |
| `output` is the observation | `WROTE output.csv ROWS 677` (a self-readback) is far more reliable than a closing sentence |
## The semantic-operator schema (five fields)
From the project's assessment (`docs/research/IDEA_ASSESSMENT_2026-09-19.md` §六), every operator must record:
| Field | Content |
|---|---|
| 适用条件 (applicability) | when the operator applies |
| 输入语义 (input semantics) | row meaning, join keys, key uniqueness, time granularity, missing-key policy |
| 操作决策 (operation decision) | fuzzy-match needed? threshold source? one-to-many allowed? unmatched handling? |
| 输出约束 (output constraints) | output granularity, key relationships, join coverage, row-count change, new missing values |
| 验证与修复 (validation & repair) | key collision, many-to-many amplification, wrong match; which doc to load per failure class |
## The eight high-error operation families
`entity alignment` · `join cardinality` · `grouped/window statistics` · `time alignment` · `missingness` · `unit handling` · `train/test transformation` · `artifact validation`
These are the project's **prototype scope** (6–10 families), not an optimal-granularity claim. The point is to pick families that *actually* produce cross-task errors — not "can't write groupby" but "used the wrong group granularity", "duplicated rows", "mixed units", "misaligned time window", "leaked full-data preprocessing into train/test".
## Three concepts that must stay separate
| Concept | Role | Do NOT inherit blindly |
|---|---|---|
| LOTUS-style semantic operator | NL-described data transform, executed by a planner | naming a skill `sem_join` gives no guarantee of LOTUS's semantics |
| Task-capability label (AgenticDataBench skills) | annotates/organizes tasks by operation mode | a classification label ≠ a loadable runtime skill |
| SKILL.md runtime skill | provides flow/knowledge/code/resources at runtime | this is the carrier; value comes from its *mechanism* |
## Mental model to hold
1. **Trajectory is evidence, operator is hypothesis.** Evidence is noisy (single-run, ±0.125); hypothesis must be gated on held-out tasks.
2. **A join's row-count change is not an error by itself** — one-to-many may be exactly what the task asks. The checker must compare against the *pre-bound* expected relationship, and report "unverified" when there is no evidence.
3. **Operators carry constraints, not answers.** A reusable decision constraint survives task change; a pasted answer does not.
4. **"Invocation count" and "effectiveness" are different measurements.** Record discovery → activation → body/resource load → adoption → outcome separately.
## References
- `AgenticDataBench/docs/research/IDEA_ASSESSMENT_2026-09-19.md` (concept split, five-field schema, eight families)
- `AgenticDataBench/generator/data_classes.py` (Skill / SkillTree / FileProfile / EvalCase data shapes)
- `AgenticDataBench/testbed/traindata/_skillopt_*/…/trajectory.json` + `agent_trace.txt` + `results.jsonl` (schema)
- `AgenticDataBench/testbed/SKILLOPT_HISTORY_PATCH.md` (pipeline functions)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |