notque/vexjoy-agent / skills/process/workflow
workflow
VerifiedStructured work: multi-phase tasks, feature builds, planning, objective loops, hill climbing.
references/auto-pipeline.md
View on GitHub---
description: "Automatic pipeline generation for unrouted tasks via /do."
user-invocable: false
allowed-tools:
- Read
- Write
- Bash
- Grep
- Glob
- Agent
- Edit
context: fork
routing:
triggers:
- auto-pipeline
- ephemeral pipeline
- no route found
- unrouted task
pairs_with:
- chain-composer
- pipeline-scaffolder
- domain-research
- pipeline-test-runner
complexity: Complex
category: meta
---
# Auto-Pipeline
## Overview
This pipeline operates as the automatic fallback for `/do` when no existing route matches a non-trivial request. It classifies the task type, selects and adapts a canonical chain, executes it with phase gates, and optionally crystallizes the pattern into a permanent pipeline.
**Key principle**: ALWAYS check the pipeline catalog first. If an existing pipeline covers 70%+ of the request, route to it instead of creating a new one. This prevents duplicate pipeline fragmentation and maintenance burden.
---
## Instructions
### Phase 0: DEDUP CHECK
**Goal**: Ensure we're not duplicating an existing pipeline.
**Why this matters**: ALWAYS check the pipeline catalog first. If an existing pipeline covers 70%+ of the request, route to it instead. Duplicate pipelines fragment routing, create maintenance burden, and confuse discovery. The dedup gate is a HARD BLOCK — apply it even when the request seems "slightly different."
**Step 1**: Run task type classification:
```bash
python3 ~/.claude/scripts/task-type-classifier.py --request "{user_request}" --check-catalog ~/.claude/skills/workflow/references/auto-pipeline/references/pipeline-catalog.json --json
```
The catalog is generated by `scripts/generate-pipeline-catalog.py` and excludes
auto-pipeline itself, so this gate can never match the request back to the
pipeline that is running it. Regenerate the catalog after adding a pipeline —
a stale catalog silently weakens the HARD BLOCK below.
**Step 2**: If the classifier returns `existing_pipeline`, STOP. Route to that pipeline instead. Display:
```
===================================================================
ROUTING: Existing pipeline found
===================================================================
Request matches: {pipeline_name} (coverage: {N}%)
Routing to existing pipeline instead of creating new.
===================================================================
```
**Step 3**: If no existing pipeline matches, capture the classification result:
- `task_type`: one of 8 canonical types
- `chain`: the canonical chain for that type
- `steps`: step count
- `keywords_matched`: which keywords triggered this classification
**Gate**: Classification complete. No existing pipeline matches. Proceed to Phase 1.
### Phase 1: CHAIN SELECTION
**Goal**: Select the best canonical chain variant and extend to 8-12 steps.
**Why 8-12 steps?** Fewer than 8 steps means under-utilizing phase gates, which reduces verification points and quality. Each additional step adds a gate — a mandatory quality checkpoint. A 6-step chain that could be 10 is leaving opportunities for validation on the table. Extend all chains to this range.
**Why sequential?** Every step must complete before the next begins. No skipping, no parallel steps unless the step explicitly supports parallelism (RESEARCH, REVIEW). Phase gates enforce ordering and prevent state leakage between phases.
**Step 1**: Read `skills/process/workflow/references/chain-composer/references/canonical-chains.md` for the full canonical chain and its variants.
**Step 2**: Select the best variant based on request analysis:
- Does the request involve unstructured input? → add EXTRACT
- Does the request need multi-source research? → add COMPILE or use GATHER variant
- Does the request need audience targeting? → add GROUND
- Does the request need iterative improvement? → add REFINE
- Does the request have verifiable syntax? → add LINT
- Does the request need spec conformance? → add CONFORM
- Does the request need post-execution observation? → add MONITOR
- Does the request involve state changes? → add CHARACTERIZE before, COMPARE after
**Step 3**: Verify chain has 8-12 steps. If fewer than 8:
- Add ASSESS before any generation/execution step
- Add COMPILE between research and generation
- Add REFINE after validation
- Add SYNTHESIZE before reporting
**Step 4**: Apply operator profile gates. Chain behavior shifts based on execution context:
- Personal: remove APPROVE, PROMPT; reduce GUARD to branch-check only
- Work: add CONFORM after GENERATE; full GUARD
- CI: skip interaction steps; add NOTIFY
- Production: add SIMULATE, SNAPSHOT, APPROVE (mandatory)
**Gate**: Chain has 8-12 steps. All type compatibility rules satisfied. Proceed to Phase 2.
### Phase 2: CONTEXT CHECK (Toolkit Repo Detection)
**Goal**: Determine whether to crystallize immediately or run ephemeral.
**Toolkit repo rule**: If running in this repo (detected by `skills/process/workflow/references/auto-pipeline.md` existing in CWD, or by the installed `~/.claude/skills/workflow/references/auto-pipeline.md` when CWD is `~/.claude`), crystallize on first encounter. This repo IS the pipeline system — every pattern we extract becomes part of the toolkit. Capture the pattern immediately on the first run.
**Outside toolkit repo rule**: Wait for 3+ ephemeral executions in the same domain before crystallizing. This ensures the pattern is stable and not a one-off.
**Step 1**: Check if `skills/process/workflow/references/auto-pipeline.md` exists in CWD (indicates toolkit repo).
**Step 2**: If toolkit repo:
- Proceed to Phase 3 (CRYSTALLIZE) instead of Phase 4 (EPHEMERAL EXECUTE)
**Step 3**: If NOT toolkit repo, count prior ephemeral runs for this domain in the repo-local run log:
```bash
grep -c "ephemeral {task_type}" .claude/ephemeral-runs.log 2>/dev/null || echo 0
```
- If 3+ found: proceed to Phase 3 (CRYSTALLIZE)
- If fewer than 3: proceed to Phase 4 (EPHEMERAL EXECUTE)
**Gate**: Execution path determined (crystallize vs ephemeral). Proceed to appropriate phase.
### Phase 3: CRYSTALLIZE (Create Permanent Pipeline)
**Goal**: Create a permanent pipeline from the adapted chain and wire it into routing.
**Why parallel research in CRYSTALLIZE?** Rule 12 is validated by A/B testing — parallel research agents produce 1.40-point quality gap over sequential grep-based research. Dispatch 3-4 parallel agents instead of running searches sequentially. Sequential research is banned in RESEARCH steps.
**Why 10 steps?** This phase itself is a full pipeline: DETECT → GATHER → RESEARCH (parallel) → COMPOSE → VALIDATE → SCAFFOLD → INTEGRATE → VERIFY → REGISTER → EXECUTE. Every step gates the next. If any step fails, we roll back to ephemeral for the current request and record the failure for investigation.
**Step 1 — DETECT**: Confirm crystallization threshold is met (toolkit repo = always; other = 3+ prior runs).
**Step 2 — GATHER**: If prior ephemeral runs exist, collect their chain descriptions from `.claude/ephemeral-runs.log`.
**Step 3 — RESEARCH**: Use accumulated evidence to inform pipeline design. Dispatch 3 parallel research agents (never sequential — Rule 12):
- Agent 1: Analyze what steps worked in prior ephemeral runs
- Agent 2: Find similar existing pipelines to learn from
- Agent 3: Identify domain-specific references/scripts needed
**Step 4 — COMPOSE**: Build the final chain:
- Start from the adapted canonical chain (from Phase 1)
- Incorporate what worked in prior ephemeral runs (if any)
- Ensure 8-12 steps
- Validate type compatibility
**Step 5 — VALIDATE**: Run `python3 ~/.claude/scripts/artifact-utils.py validate-chain` against the composed chain. If validation fails, fall back to unmodified canonical chain and log the adaptation failure.
**Step 6 — SCAFFOLD**: Create the pipeline skill:
- Create `skills/process/workflow/references/{pipeline-name}.md` with full operator context
- Create `skills/process/workflow/references/{pipeline-name}/references/` if domain references are needed
- Follow the existing pipeline SKILL.md format (frontmatter + operator context + phases)
**Step 7 — INTEGRATE**: Wire into routing:
- Run `python3 ~/.claude/scripts/generate-skill-index.py` to update INDEX.json
- The toolkit skill's Routing Tables mode handles /do routing table updates
**Step 8 — VERIFY**: Confirm the new pipeline is discoverable:
- Check `skills/INDEX.json` contains the new pipeline
- Verify `skills/process/workflow/references/{name}.md` exists and parses
**Step 9 — REGISTER**: Record the crystallization event:
- Append one line to `.claude/ephemeral-runs.log` naming the new permanent pipeline
- The pipeline file itself is the durable record; the log line stops the counter re-crystallizing the same domain
**Step 10 — EXECUTE**: Route the original request through the newly created permanent pipeline.
**Gate**: Pipeline created, integrated, verified. Original request executing through permanent pipeline. If any earlier step fails, fall back to ephemeral execution for this request and record the failure.
### Phase 4: EPHEMERAL EXECUTE
**Goal**: Execute the adapted chain inline with phase gates, without persistence.
**Why save intermediate artifacts?** Context is ephemeral; files are not. Save intermediate output at each phase to session-local files so work survives context compression and session boundaries.
**Step 1**: Display the ephemeral pipeline banner:
```
===================================================================
ROUTING: [EPHEMERAL PIPELINE] {task_description}
===================================================================
Task type: {task_type}
Chain ({N} steps): {step1} → {step2} → ... → {stepN}
Profile: {profile} ({profile_description})
Note: No permanent pipeline exists for this domain.
Running ephemeral chain with phase gates.
===================================================================
```
**Step 2**: Execute each step in sequence with gating:
- For each step in the chain:
1. Announce: `[Phase {N}/{total}: {STEP_NAME}]`
2. Execute the step's action (research, compile, generate, validate, etc.)
3. Save output to session-local file: `/tmp/ephemeral-pipeline/{step_name}.md`
4. Verify gate condition is met (step must complete, output must exist)
5. Proceed to next step only after gate passes
**Step 3**: For RESEARCH steps (Rule 12 mandatory): dispatch 3-4 parallel agents. Never run grep/search commands sequentially — that is the banned pattern. Instead:
- Agent 1: Code/content analysis
- Agent 2: Usage patterns / ecosystem context
- Agent 3: Examples and reference material
- Agent 4 (optional): External documentation / API references
Collect all results in parallel, then synthesize.
**Step 4**: For REVIEW steps: dispatch 3+ parallel reviewers with different lenses (contrarian, skeptical, user advocate, etc.).
**Step 5**: For REFINE steps: iterate up to 3 times until validation passes. After 3 iterations, proceed with best effort output and log what remains unresolved.
**Step 6**: On completion, append one line to the repo-local run log so Phase 2 can count runs:
```bash
mkdir -p .claude && printf '%s ephemeral %s for %s: %s\n' "$(date -u +%F)" "{task_type}" "{domain}" "{chain_description}" >> .claude/ephemeral-runs.log
```
This enables the 3+ run threshold for crystallization outside the toolkit repo.
**Gate**: All steps executed. Output delivered. Run logged.
---
## Error Handling
### Error: "Classification returns no matches"
**Cause**: Request doesn't match any task type keywords.
**Solution**: Default to `analysis` type (broadest applicability). Log a learning about the unclassifiable request.
### Error: "Dedup gate blocks but user insists"
**Cause**: Existing pipeline matches 70%+ but user says it doesn't serve their needs.
**Solution**: User can say "create new pipeline for X, existing one doesn't cover Y." The explicit override bypasses dedup.
### Error: "Chain validation fails"
**Cause**: Type compatibility error in adapted chain.
**Solution**: Fall back to the unmodified canonical chain for that task type. Log the adaptation that failed.
### Error: "Crystallization fails mid-scaffold"
**Cause**: Pipeline creation blocked by creation gate or ADR enforcement.
**Solution**: Fall back to ephemeral execution for the current request. Record the failure for investigation.
---
## References
- [Canonical Chains](../chain-composer/references/canonical-chains.md) - 8 canonical chain templates with variants
- [Step Menu](../pipeline-scaffolder/references/step-menu.md) - 23 step families, 150+ steps, type compatibility matrix
- [Pipeline Catalog](references/pipeline-catalog.md) - Auto-generated catalog of all existing pipelines (dedup reference)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies. | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |
| meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要". | Stars: 108k | Updated | Claude CodeCodex | Verified | Install |