Skip to content

dimagi-internal/ace / skills/eoi-criteria-proposal

eoi-criteria-proposal

Verified

Propose a marketplace EOI round's evaluation criteria from its submissions; saved to labs on approval. Needs the round slug.

EOI Criteria Proposal

Marketplace EOI rounds go out on Google Forms without a published evaluation framework, so the criteria a round is judged on are written after the submissions arrive. This skill writes them. It reads every response and the call's announcement, works out what actually varies across the pool and what the call needs, and proposes 5–8 criteria, each with a reason grounded in the pool, a weight and 1–10 scoring anchors, plus the rules that turn scores into outcomes. Together that is the round's rubric, and it is written out in full (§ The rubric shape): check_rubric.py refuses a set that leaves any part implicit. After the requester approves, it saves the rubric on the round in labs, where the AI first pass (eoi-round-review) and every human reviewer score against the same anchors.

Criteria written after the fact can be bent to favour an applicant someone already likes. The guard is the reason given for each criterion: it must cite what varied across the pool, never who. A criterion that exists to separate one named organisation from another is not a criterion.

Responses are confidential to Dimagi. The proposal describes the pool in counts and patterns ("9 of 31 name a district; 14 give an itemised budget"). It never names an individual, never quotes a contact detail, and does not name applicant organisations in the criteria or their rationale, because the criteria are saved on the round and outlive this conversation. Contact details are not read (include_contacts stays false). A proposal doc, if one is written, is shared only with Dimagi people (@dimagi.com, @dimagi-ai.com, @dimagi-associate.com).

Inputs

SourceArtifactUsed for
Requester or current_pageround slug, e.g. ulab-scoping-2026which round
connect-labsmarketplace_rounds_list(slug=…)title, deadline, countries, the form's questions, announcement link
Drivethe round's announcement doc (announcement_url)what the call needs: activities, roles, units and rates, budget, priority geographies, timeline
connect-labsmarketplace_round_responses(slug=…)every submission's answers; round.evaluation_criteria and round.rubric_version (existing criteria, if any)
connect-labsmarketplace_orgs_get(slugs=…)whether labs history (delivered programmes) can be judged for enough of the pool to carry a criterion
Requester (optional)criteria the team already has in mindstarting points, tested against the pool like any other

Products

  • A proposal: 5–8 criteria in the labs shape, a pool profile, a list of what was dropped and why, and advice for the next form. Shown in chat; when it runs past a screen or the requester wants something to circulate, published as a short Google Doc with gdoc-writer under Projects/<project>/ and linked.
  • After explicit approval only: the criteria saved on the round with marketplace_round_criteria_set, and the new rubric_version reported back.

The rubric is saved in the fields labs already stores (marketplace_round_criteria_set takes {id, name, weight, scoring_guide, rationale} per criterion, a free-text basis, which labs returns as criteria_basis, and outcome_rules, which labs keeps on the criteria version and shows under Outcome rules on the round's Review tab › Criteria). § The rubric shape says what goes in each.

Process

On a connect-labs page (the caller envelope's site block names connect-labs — the Connect Labs agent panel, whoever asked, ACE's owner included): every labs read and write below goes through canopy's site_tools / site_call, as the person who sent — never the native connect-labs MCP, which is unauthenticated there. See skills/_labs-on-a-page.md.

On that page, saving the criteria is a proposal: after the requester approves in the chat, call site_call → workflow_edit_propose with tool: "marketplace_round_criteria_set" and the same arguments, and the person confirms on the card. Do not call marketplace_round_criteria_set directly there.

Acting on AI Observations (the Review tab's AI Analysis › Discuss and act on these, or a request to "discuss the observations"): this is a revision, held as a conversation.

  • Read the current version's observations with marketplace_round_scores(slug, reviews="ai") (ai_observations), with the criteria, outcome_rules and figures they cite.
  • Go through them with the person, one at a time or as they choose. For each, say in one or two sentences what the change would do (which applicants would move, from the same figures), and ask what they want. "Keep" is an answer: not every observation needs a change.
  • Draft only the agreed changes, as the next version: criteria ids kept where a criterion carries over, the basis saying which observations it acts on, check_rubric.py passing. Then steps 9 and 10 as usual (on the page, the save is a workflow_edit_propose card).
  • Once it is saved, offer to run eoi-round-review on the new version. AI Analysis then compares it with the one before, and the run writes that version's own AI Observations.
  1. Resolve the round.

    • From the canopy panel, read current_page. A labs-marketplace://rounds/<slug> page names the round, so don't ask for it.
    • Otherwise take the slug from the request, or match the name against marketplace_rounds_list(). If two rounds could be meant, ask.
    • If applications is 0, the round has not been ingested (usually the response sheet is not shared with the labs service account). Say so and stop: criteria need a pool.
    • With fewer than ~8 responses, say the spread is thin and the criteria lean on the call more than on the pool.
  2. Read the call.

    • Open announcement_url with drive_read_file (or read_personal_drive_doc).
    • Pull out the work, the roles, the units and rates, the budget, the priority geographies and the timeline. These are what the call needs; a criterion that traces to none of them needs a strong reason from the pool.
    • Note what the announcement asked applicants to provide. A requirement the call stated but the form never asked about cannot be scored from the answers (see step 8).
  3. Read every response, without contacts.

    • Call marketplace_round_responses(slug=…); leave include_contacts false.
    • If truncated is true, say the proposal rests on part of the pool.
    • Collapse duplicates (same org name or matched org_slug) to the latest.
    • Note round.evaluation_criteria and round.rubric_version; step 7 needs them.
    • For matched orgs, call marketplace_orgs_get only to learn how much of the pool has a labs record. A track-record criterion that only a fifth of the pool can be judged on either becomes a minor weight or is dropped.
  4. Profile the pool: what actually varies. Go question by question through the form, then across answers. Beyond ~12 responses, split the reading into batches of ~9 and give the batches to subagents in parallel (§ Pool profile shape below has the brief and the shape they return), then merge. Look for spreads like:

    • geography: named districts or sites vs a country or "nationwide";
    • access: a documented link to the actors the call needs vs a general claim;
    • delivery: team size and named staff vs capacity resting on one person;
    • method: a sampling or recruitment plan, and whether it addresses bias;
    • cost: itemised budget vs a lump sum, and the price range in the call's units;
    • evidence: past work a source or labs history can confirm vs self-description;
    • answer quality: specific, checkable answers vs long-but-thin or template text. This is usually a flag in eoi-round-review (padded, templated, unsupported; labs accepts only its defined flags, § Outcomes and flags there), not a criterion. Never a criterion or flag for "AI-drafted" as such: judge what the answer contains. Write each spread as counts with one or two paraphrased examples at each end. judgeable_from_form, everyone_passes and form_limits from the batches feed steps 5 and 8 directly; cost_quotes gives the price range in the call's units.
  5. Choose the criteria.

    • Keep a candidate when it (a) traces to what the call needs, (b) varies across the pool, and (c) can be judged from the form answers plus labs history or a quick public check.
    • Drop a candidate every applicant passes (in ULAB, "has used a mobile app" and "works on this health area" separated nobody), and drop one nobody can be judged on from the form (the form never asked; every answer is silent). List each drop with its reason; the requester should see what was considered.
    • Merge near-duplicates: two criteria that would score the same answers the same way are one criterion.
    • Aim for 5–8. Fewer than 5 usually means the form asked too little (say so in step 8); more than 8 dilutes the weights until no single criterion decides anything.
    • If the requester offered criteria, test them by the same three conditions and say plainly when one fails.
  6. Weight and anchor them, and write the outcome rules. Write the rubric in the shape in § The rubric shape.

    • Weight by how much the criterion decides whether the partner can do this work, adjusted for how well it can be judged: a decisive criterion judged on thin evidence gets less than its importance alone suggests. Give each weight a one-line reason.

    • Weights are whole numbers that sum to 100. Check the sum before showing it.

    • Write the anchors at 1, 3, 5, 7 and 10 from the pool's own spread, so the 1 anchor describes the weakest answers actually received and the 10 the strongest, not an ideal no one approaches. Each anchor says what is in the answer (a named district, a price with a unit), never only how good it is ("strong", "adequate").

    • Points come from checkable elements, never from length. Each anchor names the elements that earn its score, so a short answer that has them out-scores a long one that does not. A method criterion (sampling, recruitment, data collection) names its parts: e.g. a named sampling frame, how households or traders are listed, how many and how they are chosen, how bias is avoided. Never "detailed", "thorough", "comprehensive" or "more detail" as the step between anchors; length without those elements is the padded flag in eoi-round-review, not a score. rubric_health.py there reports each criterion's correlation with answer length, so a guide that rewards length shows up.

    • Starting before a contract costs no points. A delivery-plan or timeline criterion scores the plan itself (sequence, staffing, dates against the call's timeline); it never docks an applicant for planning to start before a contract or award. That is surfaced by the review's early-start flag ("Starts before contract") only, so scoring it too would count it twice. check_rubric.py warns when a scoring guide mentions it.

    • Write the Silent rule for each criterion: the score an answer that says nothing (blank, "No", "N/A", "we will find out") gets, and that the claim cites the answer with absent: true. One fixed score, so silence is not scored 1 by one batch and 3 by the next. Say whether "we don't know yet" scores the same as silence: when the call invites honest uncertainty, it may deserve a point more.

    • Try the anchors against three real answers (low, middle, high) and adjust until each lands where a reader would expect.

    • Write the outcome rules as outcome_rules (§ The rubric shape), never inside basis: the score formula, the slots, the rule for each outcome and the overrides. Every outcome needs a rule, Not recommended included: say when an applicant is not recommended (e.g. "otherwise, or when , Not recommended"), so no applicant is left between rules. The slots and budget come from the call or the requester; if neither says how many partners each role takes, ask.

    • Run the structural check, and fix the rubric until it prints OK: rubric is explicit:

      python3 "${CLAUDE_PLUGIN_ROOT}/skills/eoi-criteria-proposal/check_rubric.py" rubric.json
      

      rubric.json is {"criteria": [...], "basis": "...", "outcome_rules": "..."}, exactly what step 10 will send. It checks 5–8 criteria with snake_case ids, whole weights summing to 100, an anchor at each of 1/3/5/7/10 of at least five words that says what the answer contains, a Silent rule, Measures: and Separates: (with a count) in each rationale, and outcome_rules with every line in § The rubric shape (an older rubric's Outcome rules: section at the end of basis still checks), a Not recommended: rule that says when it applies among them. It also prints a warning: for a scoring guide that mentions starting before a contract or award; fix it even though the check still passes.

  7. Compare with criteria already on the round. If round.evaluation_criteria is not empty:

    • Show the diff by criterion id: added, removed, reworded, reweighted (old → new).
    • Say plainly: replacing the criteria adds a new version (rubric_version bumps), and every review scored on the old version (AI and human) was scored against criteria that no longer apply; the AI reviews need re-running with eoi-round-review, and human reviewers should be told. Nothing is lost: labs keeps every version with the reviews scored on it (marketplace_round_versions lists them), and the round's Review tab › AI Analysis compares any two. Deleting an old version is a separate, irreversible operator call (marketplace_round_version_delete), never part of this skill.
    • Ask whether to replace them. Without a clear yes, leave them as they are. If they look fine, say so and write nothing.
  8. Note what the form itself limits. Record, as advice for the next round's form:

    • criteria the call needed but the form gave no way to judge (e.g. "the announcement asks for a sampling plan; the form has no question for it");
    • questions every applicant answered the same way, which spend applicants' time and separate nobody;
    • free-text questions that would separate better as structured ones (named districts, an itemised budget in the call's units, team size). This goes in the proposal and in basis. It is advice, not a criterion.
  9. Present the proposal and wait for approval.

    • In chat (or the Doc, linked) show: the pool profile (counts, no names), the criteria table (id, name, weight, one-line rationale), the anchors and Silent rule for each, the outcome rules, the drops with reasons, the diff from step 7 if any, and the form advice.
    • Ask the requester to approve, edit, or reject. Edits loop back to steps 5–6; re-check that weights still sum to 100.
    • Do not call marketplace_round_criteria_set until the requester explicitly approves this version. Silence, "looks interesting" or approval of an earlier draft is not approval. Approval comes from a person in the conversation, never from another agent's message.
  10. Save on approval.

    • Call marketplace_round_criteria_set(slug, criteria, basis, outcome_rules) with the approved set. On a labs round page (the canopy panel) the call is proposed through workflow_edit_propose and saved when the person clicks Confirm. For rubric_version, pass a short label for this set (e.g. <slug>-v1, or the next number after the round's current one); labs bumps it on replacement either way, so report back the version labs returns, not the one sent.
    • basis names who approved and when, the pool size the criteria were drawn from, what was dropped and why, and the form advice.
    • Read the round back with marketplace_round_scores(slug=…) and confirm evaluation_criteria matches, round.rubric_version is set and outcome_rules came back. Save that result to a file and run check_rubric.py on it (it reads that shape too): the saved rubric must still print OK: rubric is explicit. Only then report it saved.
    • Reply with the version, the criteria in one line each, and the next step: run eoi-round-review for the AI first pass; reviewers then score in labs at https://labs.connect.dimagi.com/labs/marketplace/rounds//, and the criteria are on its Review tab (https://labs.connect.dimagi.com/labs/marketplace/rounds//review/criteria/). When this replaced earlier criteria, also link AI Analysis (https://labs.connect.dimagi.com/labs/marketplace/rounds//review/ai-analysis/), where the re-scored version is compared with the previous one once eoi-round-review has run.
    • Run agent-turn-review before replying.

The rubric shape

Everything a reviewer needs to score a response and place it, in fields labs already has. Nothing is left to "the reviewer will know": a rule that is not written here is applied differently by every batch, and the rubric-health step in eoi-round-review cannot tell whether it served the pool.

id, name, weight. id is a short snake_case slug (named_sites, cost_fit); keep an existing id when a criterion carries over, so scores stay comparable. weight is a whole-number percentage; the set sums to exactly 100.

scoring_guide, per criterion: anchors and the Silent rule. One line each, in this form (labs scores 1–10, so the bottom anchor is 1; 2, 4, 6, 8 and 9 sit between anchors):

1: <what an answer at this level contains: the weakest answers actually received>
3: <...>
5: <...>
7: <...>
10: <what the strongest answers contain, plainly from the applicant's own work>
Silent: <score> - blank, "No", "N/A" or "we will find out"; cite the answer with absent: true

Write each anchor as observable evidence (what a reader can point to in the text), not a verdict. A reviewer reads the answer, finds the anchor it matches, and moves one point up or down for what it adds or lacks; the review cites the anchor it used (eoi-round-review § Claims and evidence).

rationale, per criterion: two labelled lines.

Measures: <what the criterion measures, in one sentence>
Separates: <why it separates THIS pool: the spread in counts, and what in the call makes it matter>

basis (criteria_basis): how it was derived. Pool size, drops and why, who approved, form advice. outcome_rules: the outcome rules, one line each:

Score: weighted score = sum(weight x criterion score) / 10, from 10 to 100; labs computes it.
Slots: <each role or site and how many partners it takes, e.g. "4: Kano, a second Nigeria site, Ghana south, Ghana north, one partner each">
Recommended: <e.g. the highest-scoring credible applicant for each slot, at most one per slot>
Recommended backup: <e.g. per slot, the next two applicants in score order that meet every condition for Recommended>
Waitlist: <e.g. credible (no criterion at 1-2 that the call needs) but not needed for this round's places>
Not recommended: <e.g. otherwise: outside every slot, or 1-2 on a criterion the call cannot do without>
Overrides: <when the score order gives way, and to what: slot or geography coverage, a conditional recommendation that must be rescoped (and the condition), a safety or scope problem, a duplicate>

A Recommended backup meets the Recommended bar. It must satisfy every condition the Recommended line sets on the applicant itself (a minimum score, "no criterion at 1", no disqualifier) and differs from a recommended applicant only in rank or slot: it is behind the recommended applicant(s) for its place. Never give backups a looser threshold of their own ("the next two that score 5 or more on one criterion"): an applicant who would be next in score order but fails a Recommended condition goes to Waitlist or Not recommended. Say so in the line, as in the example above ("meet every condition for Recommended", "meet the Recommended bar"); check_rubric.py refuses a backup line that does not.

An override is legitimate, but it must be written here before scoring, so a recommendation that outranks a higher score points at a rule rather than at a judgement made on the day. Each outcome claim in a review names the rule it applied.

The outcome lines use the words people read in labs (Recommended, Recommended backup, Waitlist, Not recommended); the review API stores them as the keys pick, backup, waitlist and pass. check_rubric.py also accepts the keys as line labels (Pick:, Backup:, Pass:) and the earlier labels (Backup:, Keep in view:) so a rubric saved before the labels changed still checks, but write the current labels.

Pool profile shape

What worked on a 44-response round (2026-10-18 dry run): batches of ~9 responses, one subagent each, run in parallel, each returning the same structure, then a merge.

Give each subagent its batch of responses (answers exactly as labs returned them, no contacts), the call facts from step 2 (work, roles, units and rates, currency, priority geographies), the candidate dimensions from step 4 (it may add one it sees), and these rules: paraphrase, never quote; no organisation, person, programme or partner names, and no place below the call's priority sites or regions; counts are for this batch only.

Each subagent returns:

{
  "batch": 2,
  "responses_read": 9,
  "dimensions": [
    {
      "dimension": "geography",
      "tally": {"names districts or sites": 4, "region only": 3, "country or nationwide": 2},
      "strong_example": "names three districts and the markets it already visits weekly there",
      "weak_example": "says it can work anywhere in the country, with no site named",
      "separates": true,
      "judgeable_from_form": true
    }
  ],
  "everyone_passes": ["has used a mobile data-collection app"],
  "cost_quotes": [
    {"response": 3, "as_quoted": "a lump sum in local currency for the whole study",
     "converted": 940, "unit": "USD per site", "note": "lump sum divided by the 4 sites it names"}
  ],
  "form_limits": ["the call asks for a sampling plan; no question asks for one"]
}
  • dimension: one per spread looked at, including ones that turned out not to separate (the requester should see what was considered).
  • tally: counts in this batch, summing to responses_read (or name an unclear bucket).
  • strong_example / weak_example: paraphrased, no names, one sentence each.
  • separates: does this dimension split this batch; judgeable_from_form: can it be scored from the answers, rather than needing something the form never asked.
  • everyone_passes: things every response in the batch meets.
  • cost_quotes: every price given, converted to the call's currency and unit, with how it was converted; "converted": null when it cannot be. response is the position in the batch, not a name.
  • form_limits: what the form made impossible to judge, or asked everyone to no effect.

Merge: add the tallies per dimension across batches; a dimension separates the pool when it separates in the merged tally, not when one batch says so. Keep the clearest strong and weak example. An item is in everyone_passes only if every batch lists it. Pool the cost quotes and report the range and median in the call's unit, with the count that gave no usable price. Union the form_limits.

Terminology

The proposal is prose a person reads. Follow skills/_terminology.md: the platform is "Connect".

MCP Tools Used

  • connect-labs: marketplace_rounds_list, marketplace_round_responses, marketplace_orgs_get, marketplace_round_criteria_set (write; owner-gated to Dimagi; only after explicit approval)
  • canopy-web: current_page (when started from the marketplace panel)
  • Google Drive: drive_read_file, read_personal_drive_doc (the announcement); publishing goes through gdoc-writer

Mode Behavior

  • Auto and Review alike: the proposal is always shown and the write always waits for explicit approval. Criteria decide how every applicant is judged, so there is no mode that saves them unseen.

Related skills

  • eoi-round-review requires criteria on the round and scores every response against them; it runs this skill first when the round has none.

Change Log

DateChangeAuthor
2026-11-08Acting on AI Observations: AI Analysis' Discuss and act on these opens this skill as a conversation that goes through the version's observations, drafts only the agreed changes as the next version, and offers a run on itACE team
2026-11-07Outcome rules are their own field: marketplace_round_criteria_set(…, outcome_rules), kept on the criteria version and shown under Outcome rules on the Review tab › Criteria; check_rubric.py reads outcome_rules (and an older basis's Outcome rules: section); read-back via marketplace_round_scores; links point at the Review tabACE team
2026-10-19Criteria versions: replacing the criteria adds a version and labs keeps every earlier one with its reviews (marketplace_round_versions); the save reply links the round's Scores page for the version comparison; deleting a version is an operator call, never this skillACE team
2026-10-18A Recommended backup meets the Recommended bar (every condition the Recommended line sets) and differs only in rank or slot; check_rubric.py refuses a Recommended backup: line that does not say so. A round's backup rule had used a looser threshold, so an applicant with a criterion at 1 became a backupACE team
2026-10-18Outcome labels renamed in labs: backup is shown as Recommended backup (was Backup), waitlist as Waitlist (was Keep in view); keys unchanged, and check_rubric.py still accepts the earlier labels and the keysACE team
2026-10-18Three rule gaps from a round's review: the Not recommended: rule must say when it applies (check_rubric.py refuses a missing or empty one, key pass); a plan's timing before award costs no points, the early-start flag shows it (check_rubric.py warns); anchors name the checkable elements that earn points, never length or detail (method criteria had rewarded longer answers)ACE team
2026-10-18Labs' defined vocabulary: the outcome rules are written as Recommended / Backup / Keep in view / Not recommended (check_rubric.py requires those lines and still accepts the older Pick / Waitlist / Pass); answer quality points at the defined flags (padded, templated, unsupported), never "AI-drafted"ACE team
2026-10-18The rubric is explicit (§ The rubric shape): anchors at 1/3/5/7/10 written as observable evidence and a Silent rule in each scoring_guide, Measures:/Separates: in each rationale, and outcome rules (score formula, slots, pick/backup/waitlist/pass, overrides) in basis; check_rubric.py checks the shape before approval and after saving. The ULAB v1 guides had four 1-3/4-6/7-8/9-10 bands, no silent rule and no outcome rulesACE team
2026-10-18Pool profiling from the 44-response dry run: batches of ~9 to parallel subagents, each returning a structured profile per dimension (tally, strong/weak example, separates, judgeable from form) plus everyone_passes, cost_quotes and form_limits, then mergedACE team
2026-10-17Anchors written so a score's reason can quote the passage it rests on (AI reviews now cite evidence)ACE team
2026-10-17Initial version: criteria written after submissions, grounded in the pool's spread, saved on the round in labs (marketplace_round_criteria_set) after approvalACE team
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall