Skip to content

dimagi-internal/ace / skills/eoi-criteria-proposal

eoi-criteria-proposal

Verified

Propose a marketplace EOI round's evaluation criteria from its submissions; saved to labs on approval. Needs the round slug.

FileKindSize
SKILL.mdskill md27.7 KB
check_rubric.pyscript13.1 KB

check_rubric.py

View on GitHub
#!/usr/bin/env python3
"""Check that a proposed EOI rubric is explicit before it is saved to labs.

Usage:
    python3 check_rubric.py <rubric.json>

rubric.json   what eoi-criteria-proposal is about to pass to
              marketplace_round_criteria_set:
                {"criteria": [{id, name, weight, scoring_guide, rationale}],
                 "basis": "<how the criteria were derived>",
                 "outcome_rules": "<the outcome rules>"}
              A round as labs returns it ({"evaluation_criteria": [...],
              "criteria_basis": "...", "outcome_rules": "..."}: the
              marketplace_round_scores result, or the whole
              marketplace_round_responses result with a "round" key) also
              works, so a saved rubric can be re-checked.

The rubric lives only in fields labs already stores:

- each criterion's `scoring_guide` holds its anchors, one per line:
      1: <what an answer at this level contains>
      3: ...
      5: ...
      7: ...
      10: ...
      Silent: <what to score, and how to cite it, when the answer says nothing>
  (labs scores 1-10, so the bottom anchor is 1);
- each criterion's `rationale` holds two labelled lines:
      Measures: <what the criterion measures>
      Separates: <why it separates THIS pool, with counts>
- `outcome_rules` (labs keeps it on the criteria version, apart from the
  basis) holds the outcome rules, with one line each for Score, Slots, Recommended,
  Recommended backup, Waitlist, Not recommended and Overrides (the outcome
  labels people read; the earlier labels Backup and Keep in view, and the
  stored keys Pick, Backup, Waitlist and Pass, are accepted for the same lines).
  The Not recommended (pass) line is required like the others: a rubric that
  says when to recommend but not when to turn an applicant down leaves the
  bottom of the pool to each reviewer.
  The Recommended backup line must say that a backup meets the Recommended bar
  (e.g. "the next two applicants in score order that meet every condition for
  Recommended"): a backup differs from a recommended applicant only in rank or
  slot, so it never skips a condition the Recommended line sets (a minimum
  score, "no criterion at 1", a disqualifier).

Also warns (without failing) when a scoring guide docks points for planning
to start before a contract or award: that is the review's `early-start` flag,
and scoring it as well counts it twice.

Prints every problem and warning, then "OK: rubric is explicit" when there
are no problems. Exits 1 on problems, 2 when the file cannot be read.
Standard library only.
"""

import json
import re
import sys

ANCHORS = (1, 3, 5, 7, 10)
RATIONALE_LABELS = ("measures", "separates")
# (line label people read, the outcome key labs stores, or None when it is not an outcome).
# Earlier labels (and the keys) are accepted for the same lines, so a rubric
# saved before the labels changed still checks.
OUTCOME_LINES = (("Score", None, ()), ("Slots", None, ()), ("Recommended", "pick", ()),
                 ("Recommended backup", "backup", ("Backup",)),
                 ("Waitlist", "waitlist", ("Keep in view",)),
                 ("Not recommended", "pass", ()), ("Overrides", None, ()))
MIN_ANCHOR_WORDS = 5
MIN_PASS_RULE_WORDS = 4
MIN_CRITERIA, MAX_CRITERIA = 5, 8
# A Recommended backup rule that says the backup clears the Recommended bar:
# "meets every condition for Recommended", "meets the Recommended bar",
# "would qualify as Recommended". A negation just before it ("need not meet")
# does not count.
_REC = r"(?:recommended|a pick|pick)"
_MEETS_BAR = re.compile(
    rf"\b{_REC}(?:'s)? (?:bar|conditions?|thresholds?|requirements?|tests?)\b"
    rf"|\b(?:conditions?|bar|thresholds?|requirements?|tests?) (?:for|of) (?:being )?{_REC}\b"
    rf"|\bqualif\w* (?:as|for) {_REC}\b",
    re.IGNORECASE)
_NEGATED = re.compile(r"(?:\b(?:not|never|without|except)\b|n't\b)[^.;\n]{0,30}$", re.IGNORECASE)
_ID = re.compile(r"^[a-z][a-z0-9_]*$")
_LINE = re.compile(r"^\s*(?:[-*]\s*)?([A-Za-z]+(?: [A-Za-z]+){0,2}|\d{1,2})\s*[:=.)-]\s*(.*\S)?\s*$")
# Words that judge an answer without saying what is in it. An anchor made only
# of these can't be applied the same way by two reviewers.
_VERDICT_ONLY = {"excellent", "good", "great", "strong", "weak", "poor", "adequate",
                 "fair", "average", "ok", "okay", "acceptable", "satisfactory",
                 "very", "somewhat", "quite", "some", "answer", "response", "the", "a",
                 "an", "is", "and", "or", "with", "of"}
# A scoring guide that mentions starting before a contract or award. Timing
# before award is the review's `early-start` flag, never a criterion's points.
_EARLY_START = re.compile(
    r"\b(start|starts|starting|started|begin|begins|beginning|commenc\w*|mobili[sz]\w*|work\w*)\b"
    r"[^.\n]{0,40}?\b(before|ahead of|prior to|without|in advance of)\b[^.\n]{0,20}?"
    r"\b(contract\w*|award\w*|signing|signature|agreement)\b"
    r"|\bpre-?(contract|award)\b",
    re.IGNORECASE)


def parse_labelled(text):
    """{label (lowercased str or int): value} for 'Label: value' lines."""
    found = {}
    for line in str(text or "").splitlines():
        m = _LINE.match(line)
        if not m:
            continue
        key, value = m.group(1), (m.group(2) or "").strip()
        found[int(key) if key.isdigit() else key.lower()] = value
    return found


def parse_anchors(scoring_guide):
    """{anchor score: text} and the silent rule, or ({}, None) when the guide is
    not in the anchored shape (e.g. an older banded '1-3: ...' guide)."""
    found = parse_labelled(scoring_guide)
    anchors = {k: v for k, v in found.items() if isinstance(k, int)}
    return anchors, found.get("silent")


def is_anchored(scoring_guide):
    anchors, _ = parse_anchors(scoring_guide)
    return all(a in anchors for a in ANCHORS)


def nearest_anchors(score):
    """The anchor(s) a score sits closest to (two on a tie)."""
    best = min(abs(score - a) for a in ANCHORS)
    return [a for a in ANCHORS if abs(score - a) == best]


def outcome_rules(basis, rules=""):
    """The outcome rules as {label: rule}: from their own field, else from an older basis.

    Rubrics saved before labs kept the rules apart carry them at the end of the
    basis after a line "Outcome rules:"; both are read.
    """
    own = str(rules or "").strip()
    if own:
        m = re.search(r"outcome rules\s*:", own, re.IGNORECASE)
        return parse_labelled(own[m.end():] if m and m.start() == 0 else own)
    text = str(basis or "")
    m = re.search(r"outcome rules\s*:", text, re.IGNORECASE)
    if not m:
        return None
    return parse_labelled(text[m.end():])


def load_rubric(raw):
    # marketplace_round_responses nests the criteria in "round"; marketplace_round_scores
    # has a "round" header (slug, title, version) and the criteria and rules beside it.
    if isinstance(raw, dict) and isinstance(raw.get("round"), dict):
        inner = raw["round"]
        if "evaluation_criteria" in inner or "criteria" in inner:
            raw = {**raw, **inner}
    if isinstance(raw, list):
        return raw, "", ""
    if not isinstance(raw, dict):
        return [], "", ""
    # evaluation_criteria first: in the scores shape "criteria" is per-criterion statistics.
    criteria = raw.get("evaluation_criteria") or raw.get("criteria") or []
    basis = raw.get("basis", raw.get("criteria_basis", ""))
    return criteria or [], basis or "", raw.get("outcome_rules") or ""


def check_anchor_text(where, text):
    words = re.findall(r"[A-Za-z0-9$%'/-]+", text or "")
    if len(words) < MIN_ANCHOR_WORDS:
        return [f"{where}: anchor is {len(words)} word(s); say what the answer contains"]
    if all(w.lower() in _VERDICT_ONLY for w in words):
        return [f"{where}: anchor judges the answer without saying what is in it"]
    return []


def check_criterion(c, n):
    cid = c.get("id") if isinstance(c, dict) else None
    where = cid or f"criterion #{n + 1}"
    if not isinstance(c, dict):
        return [f"{where}: not an object"]
    problems = []
    if not cid or not _ID.match(cid):
        problems.append(f"{where}: id must be a snake_case slug")
    if not (c.get("name") or "").strip():
        problems.append(f"{where}: no name")
    w = c.get("weight")
    if isinstance(w, bool) or not isinstance(w, (int, float)) or w != int(w) or w <= 0:
        problems.append(f"{where}: weight {w!r} is not a positive whole number")

    anchors, silent = parse_anchors(c.get("scoring_guide"))
    for a in ANCHORS:
        if a not in anchors:
            problems.append(f"{where}: scoring_guide has no anchor for {a}")
        else:
            problems.extend(check_anchor_text(f"{where} anchor {a}", anchors[a]))
    extra = sorted(set(anchors) - set(ANCHORS))
    if extra:
        problems.append(f"{where}: anchors {extra} are not {'/'.join(map(str, ANCHORS))}")
    texts = [anchors[a].lower() for a in ANCHORS if anchors.get(a)]
    if len(set(texts)) != len(texts):
        problems.append(f"{where}: two anchors have the same text")
    if not silent:
        problems.append(f"{where}: scoring_guide has no 'Silent:' rule for an answer that says nothing")
    elif not re.match(r"^\s*(10|[1-9])\b", silent):
        problems.append(f"{where}: the Silent rule must start with the score a silent answer gets (1-10)")

    found = parse_labelled(c.get("rationale"))
    for label in RATIONALE_LABELS:
        if not found.get(label):
            problems.append(f"{where}: rationale has no '{label.capitalize()}:' line")
    if found.get("separates") and not re.search(r"\d", found["separates"]):
        problems.append(f"{where}: 'Separates:' gives no count from the pool")
    return problems


def warnings_for(criteria):
    """Things worth fixing that do not make the rubric implicit."""
    out = []
    for n, c in enumerate(criteria if isinstance(criteria, list) else []):
        if not isinstance(c, dict):
            continue
        if _EARLY_START.search(str(c.get("scoring_guide") or "")):
            out.append(f"warning: {c.get('id') or f'criterion #{n + 1}'}: scoring_guide mentions starting before "
                       "a contract or award; timing before award costs no points, the review's early-start "
                       "flag shows it")
    return out


def check(criteria, basis, rules_text=""):
    problems = []
    if not isinstance(criteria, list) or not criteria:
        return ["no criteria"]
    if not MIN_CRITERIA <= len(criteria) <= MAX_CRITERIA:
        problems.append(f"{len(criteria)} criteria; propose {MIN_CRITERIA}-{MAX_CRITERIA}")
    ids = [c.get("id") for c in criteria if isinstance(c, dict)]
    for dup in sorted({i for i in ids if i and ids.count(i) > 1}):
        problems.append(f"{dup}: id used twice")
    for n, c in enumerate(criteria):
        problems.extend(check_criterion(c, n))
    weights = [c.get("weight") for c in criteria if isinstance(c, dict)]
    if all(isinstance(w, (int, float)) and not isinstance(w, bool) for w in weights):
        total = sum(weights)
        if total != 100:
            problems.append(f"weights sum to {total:g}, not 100")

    rules = outcome_rules(basis, rules_text)
    if rules is None:
        problems.append("no outcome rules: send them as outcome_rules")
    else:
        for label, key, earlier in OUTCOME_LINES:
            names = [label, *earlier] + ([key] if key else [])
            rule = next((rules[n.lower()] for n in names if rules.get(n.lower())), None)
            if not rule:
                problems.append(f"outcome rules have no '{label}:' line"
                                + (f" (key {key}): say who is not recommended, e.g. 'otherwise, or when "
                                   "<disqualifier>'" if key == "pass" else ""))
            elif key == "pass" and len(rule.split()) < MIN_PASS_RULE_WORDS:
                problems.append("the 'Not recommended:' rule must say when it applies, e.g. 'otherwise, "
                                "or when <disqualifier>'")
            elif key == "backup" and not meets_recommended_bar(rule):
                problems.append("the 'Recommended backup:' rule must say a backup meets the Recommended bar, "
                                "e.g. 'per slot, the next two applicants in score order that meet every "
                                "condition for Recommended'")
    return problems


def meets_recommended_bar(rule):
    """True when a backup rule says the backup clears every Recommended condition."""
    return any(not _NEGATED.search(rule[:m.start()]) for m in _MEETS_BAR.finditer(rule))


def main(argv):
    if len(argv) != 2:
        print("usage: python3 check_rubric.py <rubric.json>", file=sys.stderr)
        return 2
    try:
        with open(argv[1], encoding="utf-8") as fh:
            raw = json.load(fh)
    except (OSError, ValueError) as exc:
        print(f"cannot read rubric {argv[1]}: {exc}", file=sys.stderr)
        return 2
    criteria, basis, rules_text = load_rubric(raw)
    problems = check(criteria, basis, rules_text)
    for w in warnings_for(criteria):
        print(w)
    if problems:
        for p in problems:
            print(p)
        print(f"{len(problems)} problem(s)")
        return 1
    print("OK: rubric is explicit")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall