Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/prompt-regression-gate

prompt-regression-gate

Verified

Gate prompt, model, tool-schema, or packer changes with paired runs on a frozen eval set, slice-level regression checks, relative baselines, and explicit go/no-go thresholds. Use before merging prompt PRs or swapping models.

FileKindSize
SKILL.mdskill md14.6 KB
gemini.mdmarkdown3.0 KB
gpt.mdmarkdown3.4 KB
---
title: "Prompt Regression Gate AI Skill Guide (GPT & Codex)"
description: "Operational skill for OpenAI GPT and Codex to wire paired prompt regression CI: hashes, baseline artifacts, relative thresholds, slice hard-fails, and rollback."
category: "AI Workflows / Prompt CI"
tags: ["prompt-regression", "ci", "baseline", "promptfoo", "gpt-codex", "eval-gate"]
---

# Prompt Regression Gate AI Skill Guide (GPT & Codex)

## Overview & Engine Architecture

GPT/Codex acts as a Principal LLM CI Engineer: implement **path-filtered workflows**, **hash manifests**, **baseline artifact promotion**, and **exit-code gates** so prompt/model PRs cannot merge on vibes.

```
┌─────────────────────────────────────────────────────────────┐
│  PR touches prompts/** → run suite → compare to main        │
│  baseline → fail on aggregate/slice/P0 delta → PR comment   │
└─────────────────────────────────────────────────────────────┘
```

---

## Operational Capabilities & Agent Directives

1. Store prompts as files; compute `prompt_hash` over system+template+tool JSON.
2. Commit or artifact-store `baseline.jsonl` from **main** only; never from the PR under test.
3. Implement relative compare (`max_aggregate_drop`, `p0_allowed_regressions=0`).
4. Prefer Promptfoo/Evalgate/custom runner + deterministic scorers; pin judge models.
5. On main merge, job refreshes baseline; document judge/dataset upgrades as explicit re-baselines.

---

## Production: GitHub Actions fragment

```yaml
name: prompt-regression-gate
on:
  pull_request:
    paths: ["prompts/**", "evals/**", "src/llm/**"]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run candidate eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/run_eval.py --prompt prompts/support.system.md --out candidate.jsonl
      - name: Fetch main baseline
        run: |
          git fetch origin main
          git show origin/main:evals/baselines/support.jsonl > baseline.jsonl
      - name: Compare
        run: python scripts/prompt_gate.py baseline.jsonl candidate.jsonl
```

Wire `prompt_gate.py` from the canonical `SKILL.md` example. Optionally `npx promptfoo eval -o results.json` then convert to per-case JSONL for the same comparator.

---

## Technical Troubleshooting Matrix

| CI symptom | Fix |
| :--- | :--- |
| Flaky gate | temp=0, cache keys include prompt+dataset hashes, raise tolerance slightly |
| Always fails after dataset edit | Bump dataset version; regenerate baseline on main |
| Cost spike | Path filters; shard suite; smoke subset on draft PRs |
| False GO | Missing P0 slice tags on cases |

---

## Best Practices

1. Majority-vote 3 runs for borderline stochastic tasks when budget allows.
2. Publish delta table as PR comment (upsert one comment).
3. Track cost tokens alongside quality in the same artifact.
4. Schema/prompt contract changes: update fixtures in the same commit.

---

## Agent Operational Directive

> **MANDATORY**: Automate paired baseline comparison with hashed provenance. Block merges on P0 and aggregate regressions. Refresh baselines only from main. Mark skipped runs as untested, not GO.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall