alivirgo/Major-AI-Skills / plugins/mas-ai-workflows/skills/llm-json-contract-check
llm-json-contract-check
VerifiedValidate AI-generated JSON against schema and business rules, separating refusals, truncation, parse failures, schema violations, and domain invariant breaches. Use before any tool call, DB write, or external side effect consumes model output.
gpt.md
View on GitHub---
title: "LLM JSON Contract Check AI Skill Guide (GPT & Codex)"
description: "Operational skill for OpenAI GPT and Codex to implement strict structured outputs, layered JSON contract validators, repair/retry loops, and CI fixtures before side effects."
category: "AI Workflows / Structured Outputs"
tags: ["json-schema", "structured-outputs", "zod", "pydantic", "validation", "gpt-codex"]
---
# LLM JSON Contract Check AI Skill Guide (GPT & Codex)
## Overview & Engine Architecture
GPT/Codex acts as a Principal API Contract Engineer: wire **strict `json_schema` / tool parameters**, generate schemas from Zod/Pydantic, and ship **CI fixtures** that prove domain rejects (e.g. negative quantity) never reach side effects.
```
┌─────────────────────────────────────────────────────────────┐
│ Pydantic/Zod model → JSON Schema (strict) → API call │
│ → refusal/finish check → validate → domain → side effect │
└─────────────────────────────────────────────────────────────┘
```
---
## Operational Capabilities & Agent Directives
1. Prefer OpenAI `response_format` with `strict: true` (or strict function tools) over prompt-only JSON.
2. Check `refusal` and incomplete `finish_reason` before `JSON.parse` / SDK `.parse()`.
3. Keep unsupported constraints (`pattern`, numeric min/max, maxItems) in application validators.
4. Implement bounded repair: feed `validator.errors` back once or twice; then fallback.
5. Emit `reject_layer` metrics and `schema_hash` on every structured call path.
---
## Production example: OpenAI + Pydantic
```python
"""Strict structured output with post-domain gate."""
from __future__ import annotations
from typing import Literal, Optional
from pydantic import BaseModel, Field, ValidationError, model_validator
class Order(BaseModel):
sku: str
quantity: int
currency: Literal["USD", "EUR", "GBP"]
note: Optional[str] = None
@model_validator(mode="after")
def qty_positive(self) -> "Order":
if self.quantity <= 0:
raise ValueError("quantity must be > 0")
return self
# Pseudo-call:
# completion = client.chat.completions.parse(..., response_format=Order)
# if getattr(completion.choices[0].message, "refusal", None): handle_refusal()
# order = completion.choices[0].message.parsed
# if order is None: handle_incomplete()
# submit_order(order) # only after parse + domain validators succeeded
```
### CI fixture pattern
```python
def test_negative_quantity_rejected():
raw = '{"sku":"SKU-1","quantity":-3,"currency":"USD"}'
# parse JSON OK → schema OK → domain FAIL
result = check_contract(raw_text=raw, schema=ORDER_SCHEMA)
assert result.ok is False and result.layer.value == "domain"
```
---
## Technical Troubleshooting Matrix
| Signature | Fix in code |
| :--- | :--- |
| Schema 400 from API | Remove unsupported keywords; inline `$ref`; flatten depth |
| `.parse()` throws on refusal | Branch on `refusal` first |
| Retries always fire | Simplify enums; split multi-stage extraction |
| Extra keys in prod | Enable strict / `additionalProperties: false` |
---
## Best Practices
1. Single model class → schema export → runtime validate.
2. Log repair strategy names (fence strip vs generative retry) separately.
3. Never coerce `sku` or money fields in the parse layer.
4. Version schemas next to prompts for regression gates.
---
## Agent Operational Directive
> **MANDATORY**: Use strict structured outputs when available. Validate schema and domain before side effects. Classify refusal/truncation correctly. Bound retries. Ship negative-path fixtures.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |