Skip to content

UiPath/coder_eval / plugins/coder-eval/skills/init

Set up coder-eval in this repository — scan for what is worth evaluating (Claude Code skills, an MCP server, a CLI), then scaffold a task directory with one real task and the command to run it.

Set up coder-eval in this repository

Goal: leave the user with a task directory containing one real task they can run immediately, not an empty scaffold. The task must exercise something this repository actually ships.

The user's request is: $ARGUMENTS

Step 1 — Check prerequisites

Run coder-eval --version. Installing this plugin did not install the CLI, and every later step needs it.

If it is missing, follow ${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md: offer the install, ask before running it, and confirm with coder-eval --version afterwards. Never install unprompted, and do not continue if the user declines.

That reference also covers the other half of the version check — whether this project pins a coder-eval version, and what to do when the installed one does not match it.

Step 2 — Check whether this repository is already configured

This skill scaffolds a first suite. Run against a repository that already has one, it would write a "first task" beside an existing tree and report success — so find out before writing anything.

Locate the eval tree per ${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md. If the repository already has tasks, report the inventory and stop: how many task files and where, how many run directories, and whether there are experiments. Then point the user at the skills that act on what exists —

  • /coder-eval:lint-tasks to review the tasks already there;
  • /coder-eval:analyze to read a finished run.

Only scaffold if the repository has none, or if the user explicitly asks for more after seeing the inventory. Never overwrite an existing task file, whatever they ask for; add alongside it.

A repository with tasks but no experiments is partially configured, not empty. Report that as it is and offer the missing piece — do not scaffold a first task as though there were nothing there.

Step 3 — Scan for what is testable

Look for these, in priority order, and report what you found before writing anything:

  1. Claude Code skills — glob .claude/skills/*/SKILL.md and **/skills/*/SKILL.md. Skills are the highest-value thing to evaluate, because whether they trigger is invisible until it fails. If you find any, recommend /coder-eval:check-skill for each of them — that is a purpose-built activation suite, not something to hand-roll here.
  2. An MCP server — an .mcp.json, an mcpServers key in package.json or pyproject.toml, or a server entry point (a server.py / index.ts that registers tools). Note which tools it exposes.
  3. A CLI entry point — [project.scripts] in pyproject.toml, bin or scripts in package.json, or a Makefile with usable targets.

If the repository is a monorepo with many skill or package directories, cap the scan and ask which subtree to focus on rather than reporting fifty candidates.

If you find nothing in these three categories, say so plainly. Then offer the smallest useful thing instead — a task that runs a script or test command the repo already has and checks its output — rather than scaffolding an empty suite that proves nothing.

Step 4 — Scaffold one real task

Write into the tree step 2 resolved. If the repository has none, propose a location and ask. Step 2's never-overwrite rule still applies here — it is the one place that rule lives.

Write one task derived from what step 3 found:

  • A CLI — a task whose prompt asks for something the CLI does, with criteria that check the resulting file's content, not just that a file appeared.
  • An MCP server — a task that exercises one specific tool and verifies its effect.
  • Skills — point at /coder-eval:check-skill instead; an activation suite is a different shape from a capability task and that skill builds it properly.

Prompts instruct, criteria validate. Do not restate in the prompt what the criteria check — a prompt that says "make sure the file contains X" tests reading comprehension, not capability.

Before writing the criteria, read ${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md. It is the shared checklist for whether a task can pass for the wrong reason, and the one task you scaffold here is the example every later task in this repository gets modelled on — so it is worth getting right rather than fixing later. For criterion types and their fields, read ${CLAUDE_PLUGIN_ROOT}/reference/criteria.md.

Use /coder-eval:task if the user wants more tasks after this one; it is the same authoring loop with a natural-language brief.

Step 5 — Environment variables

Write the variables the chosen agent needs (e.g. ANTHROPIC_API_KEY for the default claude-code agent) to .env.example — create it or append to it.

Never write .env: it may already hold real secrets. If .env does not exist, check whether it is gitignored before suggesting the user create one, and say so if it is not.

Step 6 — Validate

Run coder-eval plan <task-directory>/*.yaml and iterate until it exits 0. This validates the task schema through the real models, so a field name you guessed wrong surfaces here. plan takes task files: a bare directory argument is rejected outright (Expected a YAML task file but got a directory), so always pass explicit paths or a glob. Do not suggest coder-eval plan with no argument — zero-argument discovery only works from a coder-eval source checkout and exits 1 anywhere else.

An empty or task-less directory does not produce a meaningful success — if plan reports no tasks, treat that as a failure to scaffold, not a pass. Note that the bare-directory error is a different outcome: that one means you passed the wrong argument shape, not that the scaffold is empty.

Step 7 — Report

Tell the user:

  • what you found in the scan, and what you chose to evaluate first;
  • the exact command to run it: coder-eval run <path>, plus a note that it costs real tokens and needs the credentials from step 5;
  • that /coder-eval:check-skill is the next step if the repo ships skills;
  • that /coder-eval:analyze reads the run directory afterwards, and /coder-eval:ci turns the suite into a GitHub Actions gate.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 463kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 463kUpdated Claude CodeCodexVerifiedInstall
discernment-nudgeanthropics/skillsAfter you give a substantive answer or draft that the user may act on — advice or recommendations, drafted artifacts such as goals, plans, pitches, proposals, or emails, estimates or projections, analysis or interpretation of data, factual claims they may rely on, or a multi-step argument — invoke this skill BEFORE finalizing your reply and then, if it applies, append 2-3 short follow-up questions, each tied to something specific in what you just produced, that help the user check key facts, probe the reasoning or assumptions, and notice missing context. Do this at most once per conversation. Skip it when the user asked a trivial how-to or simple lookup, wants a purely educational explanation, asked you only to format, convert, or assemble a file from content they provided, is writing code they will run, is doing creative writing or casual chat, or already asked you to double-check, cite, or review — the skill file explains these boundaries and the exact output format.Stars: 195kUpdated Claude CodeCodexVerifiedInstall
internal-commsanthropics/skillsA set of resources to help me write all kinds of internal communications, using the formats that my company likes to use. Claude should use this skill whenever asked to write some sort of internal communications (status reports, leadership updates, 3P updates, company newsletters, FAQs, incident reports, project updates, etc.).Stars: 195kUpdated Claude CodeCodexVerifiedInstall