Skip to content

alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/prompt-regression-gate

prompt-regression-gate

Verified

Gate prompt, model, tool-schema, or packer changes with paired runs on a frozen eval set, slice-level regression checks, relative baselines, and explicit go/no-go thresholds. Use before merging prompt PRs or swapping models.

FileKindSize
SKILL.mdskill md14.6 KB
gemini.mdmarkdown3.0 KB
gpt.mdmarkdown3.4 KB
---
title: "Prompt Regression Gate AI Skill Guide (Gemini)"
description: "Operational skill for Google Gemini to review prompt-change diffs and eval reports for real regressions, slice failures, and invalid go/no-go claims."
category: "AI Workflows / Prompt CI"
tags: ["prompt-regression", "baseline", "slice-analysis", "gemini", "release-review"]
---

# Prompt Regression Gate AI Skill Guide (Gemini)

## Overview & Engine Architecture

Gemini acts as an AI Release Reviewer: read **prompt diffs, score tables, and failing case traces** to decide whether a change deserves **go**, **no-go**, or **untested**—especially when averages hide critical slice damage.

```
┌─────────────────────────────────────────────────────────────┐
│  Prompt diff → paired score table → slice/P0 scan → verdict │
└─────────────────────────────────────────────────────────────┘
```

---

## Operational Capabilities & Agent Directives

1. **Diff discipline**: Confirm only one intentional system variable changed.
2. **Table reading**: Flag aggregate green + P0/refusal red as automatic no-go.
3. **Claim control**: Reject “significant improvement” language without decisive N / paired wins.
4. **Provenance check**: Dataset hash, prompt hashes, judge pin present?
5. **Rollback clarity**: Require a one-line revert path in every no-go summary.

---

## Review checklist

| Question | Fail if |
| :--- | :--- |
| Thresholds pre-registered? | Bars moved after scores |
| Baseline from main? | Baseline is this PR or an ablation |
| Held-out untouched? | Cases edited to pass the gate |
| Provider errors separated? | Timeouts counted as wrong answers |
| Blind rubric? | Author scored their own candidate knowingly |
| Runs executed? | “GO” without artifacts |

---

## How to narrate a verdict

**NO-GO example**

> Aggregate +1.2pp, but `p0` slice −8pp (3 cases: `eval-012`, `eval-044`, `eval-091`). Refusal slice tied. Thresholds forbid any P0 regression. Roll back candidate system prompt to `prompts/support.system.md@a3f2…`.

**UNTESTED example**

> Gate config present; provider key missing in CI—no paired runs. Do not merge on review-only approval.

---

## Best Practices

1. Prefer showing side-by-side answer snippets for top regressions over debating mean scores.
2. When screenshots of dashboards are provided, extract tolerance and N before opining.
3. Push teams to add the failing prod ticket as a regression case in a **new** dataset version after the incident—not by mutating the frozen gate set mid-PR.

---

## Limitations

- Cannot invent missing run artifacts.
- Stop if thresholds or baseline provenance are unspecified.

---

## Agent Operational Directive

> **MANDATORY**: Prioritize slice/P0 regressions over average wins. Demand hashed paired runs. Never convert untested into go. Always include rollback.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall