moonlight-lupin/agent-skills / mlops/model-compare
model-compare
FlaggedBlind side-by-side multi-model comparison. Send one prompt to 2-4 models simultaneously, present responses anonymously (Model A / B / C / D), let the user pick a winner, then reveal identities and show which model won. Supports custom evaluation criteria, synthesis of responses, and vote history logging. Trigger when the user says "compare models", "test these models", "which model is better for", "A/B test", "blind comparison", "model evaluation", or wants to see how different AI models handle the same prompt. Can also be used for prompt engineering — testing how different models interpret the same instructions. Supports reasoning-effort A/B via provider:@effort:model_id specs.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 59.5 KB |
| references/model-behavior.md | markdown | 33.1 KB |
| references/provider-tool-support.md | markdown | 5.5 KB |
| references/providers.json | config | 2.8 KB |
| requirements-dev.txt | markdown | 230 B |
| scripts/aggregate_battery.py | script | 8.0 KB |
| scripts/compare.pyexecutable | script | 77.0 KB |
| scripts/embedding_compare.pyexecutable | script | 11.9 KB |
| scripts/provider_health.py | script | 9.7 KB |
| tests/__init__.py | script | 0 B |
| tests/test_compare.py | script | 29.0 KB |
| tests/test_embedding_compare.py | script | 10.4 KB |
| tests/test_providers_snapshot.py | script | 2.8 KB |
tests/test_providers_snapshot.py
View on GitHub#!/usr/bin/env python3
"""Keep references/providers.json honest.
The canonical provider registry is `PROVIDERS` in scripts/compare.py. The JSON
snapshot is a human-facing convenience; this test fails if the two drift apart
(a provider added/removed/renamed in code but not the snapshot, or a mismatched
env var / endpoint / cost). It also validates the snapshot's shape. Pure stdlib.
"""
import json
import os
import sys
from pathlib import Path
SKILL_DIR = Path(__file__).resolve().parent.parent
SCRIPTS_DIR = SKILL_DIR / "scripts"
sys.path.insert(0, str(SCRIPTS_DIR))
import compare # noqa: E402
SNAPSHOT_PATH = SKILL_DIR / "references" / "providers.json"
def _load_snapshot():
return json.loads(SNAPSHOT_PATH.read_text(encoding="utf-8"))
def test_snapshot_is_valid_json_with_required_keys():
snap = _load_snapshot()
for key in ("snapshot_date", "routing_order", "providers"):
assert key in snap, f"providers.json missing top-level key: {key}"
assert isinstance(snap["providers"], dict) and snap["providers"]
def test_provider_set_matches_code():
snap = _load_snapshot()
assert set(snap["providers"]) == set(compare.PROVIDERS), (
"providers.json is out of sync with PROVIDERS in compare.py — "
f"snapshot={sorted(snap['providers'])} code={sorted(compare.PROVIDERS)}"
)
def test_routing_order_covers_every_provider_exactly_once():
snap = _load_snapshot()
assert sorted(snap["routing_order"]) == sorted(compare.PROVIDERS)
assert len(snap["routing_order"]) == len(set(snap["routing_order"]))
def test_each_provider_env_endpoint_and_cost_match_code():
snap = _load_snapshot()
for name, p in snap["providers"].items():
cfg = compare.PROVIDERS[name]
assert p["env"] == cfg["key_env"], f"{name}: env mismatch"
assert p["endpoint"] == cfg["base_url"], f"{name}: endpoint mismatch"
expected_cost = "paid" if cfg.get("paid") else "free"
assert p["cost"] == expected_cost, f"{name}: cost mismatch"
def test_each_provider_lists_representative_example_models():
snap = _load_snapshot()
for name, p in snap["providers"].items():
assert isinstance(p.get("example_models"), list) and p["example_models"], (
f"{name}: example_models must be a non-empty list"
)
assert all(isinstance(m, str) and m for m in p["example_models"])
def test_snapshot_carries_no_hardcoded_model_counts():
"""The whole point of the snapshot is to avoid drift-prone counts."""
raw = SNAPSHOT_PATH.read_text(encoding="utf-8")
# No "<provider>: 121 models"-style integers in the model fields.
for name, p in _load_snapshot()["providers"].items():
assert "count" not in p and "model_count" not in p, f"{name}: drop model counts"
assert raw # sanity
if __name__ == "__main__":
import pytest
pytest.main([__file__, "-v"])
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |