Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/ai-evaluation-dataset

ai-evaluation-dataset

Verified

Build versioned JSONL evaluation datasets for AI workflows with golden/regression/holdout layers, leakage checks, rubrics, and provenance hashes. Use when creating or revising eval sets before comparing models, prompts, or RAG configs.

FileKindSize
SKILL.mdskill md15.1 KB
gemini.mdmarkdown4.3 KB
gpt.mdmarkdown4.8 KB
---
title: "AI Evaluation Dataset AI Skill Guide (GPT & Codex)"
description: "Operational skill for OpenAI GPT and Codex to author versioned JSONL eval datasets, manifests, content hashes, leakage checks, and CI-bound result rows."
category: "AI Workflows / Evaluation"
tags: ["evaluation-dataset", "golden-set", "jsonl", "versioning", "leakage", "gpt-codex", "ci"]
---

# AI Evaluation Dataset AI Skill Guide (GPT & Codex)

## Overview & Engine Architecture

GPT/Codex acts as a Principal Evaluation Platform Engineer: generate **schemas, freeze scripts, CI gates, and bridge reports** so every metric row binds `dataset_hash`, `system_hash`, and `judge_hash`.

```
┌─────────────────────────────────────────────────────────────┐
│                 Dataset Automation Pipeline                 │
│  JSONL cases → validate schema → group-split check → hash   │
│  → manifest freeze → eval job → result row + artifacts      │
└─────────────────────────────────────────────────────────────┘
```

---

## Operational Capabilities & Agent Directives

1. Emit `evals/data/vN/{golden,regression,holdout}.jsonl` + `manifest.json`.
2. Implement validators: required fields, unique `id`, paraphrase co-location, PII heuristics.
3. Wire CI to fail if `HASH` file ≠ recomputed content hash.
4. On label PRs, auto-generate a bridge job: same `system_hash` on old vs new dataset versions.
5. Keep judge/assertion code versioned beside cases; never inline unscored “vibes.”

---

## Production Python: manifest freeze CLI sketch

```python
"""freeze_dataset.py — validate, hash, write manifest."""
from __future__ import annotations

import argparse
import json
from datetime import date
from pathlib import Path

# Reuse helpers from SKILL.md: content_hash, load_jsonl, assert_paraphrase_groups_colocated

REQUIRED = {"id", "split", "input", "expected_behavior", "forbidden_behavior", "provenance"}


def validate_rows(rows: list[dict]) -> None:
    ids: set[str] = set()
    for row in rows:
        missing = REQUIRED - row.keys()
        if missing:
            raise SystemExit(f"{row.get('id')}: missing {missing}")
        if row["id"] in ids:
            raise SystemExit(f"duplicate id: {row['id']}")
        ids.add(row["id"])


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--dir", type=Path, required=True)
    ap.add_argument("--version", required=True)
    ap.add_argument("--description", default="")
    args = ap.parse_args()

    files = sorted(args.dir.glob("*.jsonl"))
    rows: list[dict] = []
    for f in files:
        rows.extend(json.loads(line) for line in f.read_text(encoding="utf-8").splitlines() if line.strip())
    validate_rows(rows)
    # assert_paraphrase_groups_colocated(rows)
    # digest = content_hash(files)
    digest = "sha256:COMPUTE"
    manifest = {
        "dataset_version": args.version,
        "created_at": date.today().isoformat(),
        "description": args.description,
        "files": [f.name for f in files],
        "case_count": len(rows),
        "content_hash": digest,
    }
    (args.dir / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
    (args.dir / "HASH").write_text(digest + "\n", encoding="utf-8")
    print(manifest)


if __name__ == "__main__":
    main()
```

### CI gate sketch

```yaml
# .github/workflows/eval-dataset.yml (fragment)
- name: Verify dataset hash
  run: |
    python scripts/freeze_dataset.py --dir evals/data/v3 --version v3 --description "check"
    git diff --exit-code evals/data/v3/HASH evals/data/v3/manifest.json
- name: Run eval
  run: python scripts/run_eval.py --dataset-version v3 --fail-under 0.85
```

---

## Technical Troubleshooting Matrix

| Signature | Check | Automation |
| :--- | :--- | :--- |
| Unstable hashes | Key order / whitespace | Canonical `sort_keys` JSON lines |
| Bridge missing | Label PR without dual run | Require `bridge.json` artifact |
| Few-shot leak | Overlap with prompt bank | Near-dup filter job |
| Schema drift | Extra fields break consumers | JSON Schema + `--strict` |

---

## Best Practices

1. Store large contexts by content hash sidecar; keep JSONL rows readable.
2. Export Promptfoo/DeepEval adapters from the same JSONL source of truth.
3. Redact before commit; scan for API keys and emails in CI.
4. Record `corpus_hash` for RAG jobs so corpus moves do not look like model wins.

---

## Agent Operational Directive

> **MANDATORY**: Automate immutability (hash files), group-aware split checks, and hash-bound result rows. Refuse in-place gold edits. Generate bridge comparisons whenever the dataset version bumps.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall