moonlight-lupin/agent-skills / mlops/model-compare
model-compare
FlaggedBlind side-by-side multi-model comparison. Send one prompt to 2-4 models simultaneously, present responses anonymously (Model A / B / C / D), let the user pick a winner, then reveal identities and show which model won. Supports custom evaluation criteria, synthesis of responses, and vote history logging. Trigger when the user says "compare models", "test these models", "which model is better for", "A/B test", "blind comparison", "model evaluation", or wants to see how different AI models handle the same prompt. Can also be used for prompt engineering — testing how different models interpret the same instructions. Supports reasoning-effort A/B via provider:@effort:model_id specs.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 59.5 KB |
| references/model-behavior.md | markdown | 33.1 KB |
| references/provider-tool-support.md | markdown | 5.5 KB |
| references/providers.json | config | 2.8 KB |
| requirements-dev.txt | markdown | 230 B |
| scripts/aggregate_battery.py | script | 8.0 KB |
| scripts/compare.pyexecutable | script | 77.0 KB |
| scripts/embedding_compare.pyexecutable | script | 11.9 KB |
| scripts/provider_health.py | script | 9.7 KB |
| tests/__init__.py | script | 0 B |
| tests/test_compare.py | script | 29.0 KB |
| tests/test_embedding_compare.py | script | 10.4 KB |
| tests/test_providers_snapshot.py | script | 2.8 KB |
references/providers.json
View on GitHub{
"snapshot_date": "2026-08",
"note": "Curated snapshot of provider config + a few representative models per provider. Intentionally carries NO live model COUNTS — those drift constantly and rot the docs. For the current, full inventory run `python3 scripts/compare.py --list-models <provider>` (needs that provider's API key); this file and `--list-models` are the source of truth for model inventories, while SKILL.md stays conceptual. The provider set here is kept in sync with `PROVIDERS` in scripts/compare.py by tests/test_providers_snapshot.py.",
"routing_order": ["ollama-cloud", "nvidia", "openrouter", "deepseek", "upstage"],
"providers": {
"ollama-cloud": {
"env": "OLLAMA_API_KEY",
"cost": "free",
"endpoint": "https://ollama.com/v1/chat/completions",
"families": ["GLM", "Qwen", "Kimi", "Gemini", "Gemma"],
"example_models": ["glm-5.2", "kimi-k2.5", "qwen3-coder:480b", "gemini-3-flash-preview", "gemma3:27b", "deepseek-v3.2"]
},
"nvidia": {
"env": "NVIDIA_API_KEY",
"cost": "free",
"endpoint": "https://integrate.api.nvidia.com/v1/chat/completions",
"families": ["Yi", "Llama", "Nemotron"],
"example_models": ["meta/llama-3.1-70b-instruct", "meta/llama-3.3-70b-instruct", "01-ai/yi-large"],
"verify_alive": "NVIDIA's /v1/models lists many entries but most return 404/410 when called — verify a model with a simple completion first. See references/provider-tool-support.md."
},
"openrouter": {
"env": "OPENROUTER_API_KEY",
"cost": "paid",
"endpoint": "https://openrouter.ai/api/v1/chat/completions",
"families": ["Claude", "GPT", "Gemini", "DeepSeek", "Llama", "Qwen", "Mistral"],
"example_models": ["anthropic/claude-sonnet-4.6", "anthropic/claude-opus-4.8", "openai/gpt-4o", "google/gemini-2.5-pro", "deepseek/deepseek-r1"],
"confirm_before_use": "Paid, per-token. Confirm with the user before any OpenRouter call."
},
"deepseek": {
"env": "DEEPSEEK_API_KEY",
"cost": "paid",
"endpoint": "https://api.deepseek.com/v1/chat/completions",
"families": ["DeepSeek"],
"example_models": ["deepseek-chat", "deepseek-reasoner"],
"notes": "Reasoning model. max_tokens=32768 and reasoning_effort=low set in payload_extra to avoid run-away reasoning loops."
},
"upstage": {
"env": "UPSTAGE_API_KEY",
"cost": "paid",
"endpoint": "https://api.upstage.ai/v1/solar/chat/completions",
"families": ["Solar"],
"example_models": ["solar-pro4"],
"confirm_before_use": "Paid, per-token. Confirm with the user before any Upstage call.",
"notes": "Upstage Solar direct API; solar-pro4 confirmed live 2026-09-02. max_tokens=16384. Tool calling via the OpenAI tools param is unverified — check before --mode tools runs. See references/model-behavior.md."
}
}
}Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |