alexandervashurin/semaphore / .claude/skills/gstack
gstack
FlaggedFast headless browser for QA testing and site dogfooding. Navigate any URL, interact with elements, verify page state, diff before/after actions, take annotated screenshots, check responsive layouts, test forms and uploads, handle dialogs, and assert element states. ~100ms per command. Use when you need to test a feature, verify a deployment, dogfood a user flow, or file a bug with evidence. gstack also includes development workflow skills. When you notice the user is at these stages, suggest the appropriate skill: - Brainstorming a new idea → suggest /office-hours - Reviewing a plan (strategy) → suggest /plan-ceo-review - Reviewing a plan (architecture) → suggest /plan-eng-review - Reviewing a plan (design) → suggest /plan-design-review - Creating a design system → suggest /design-consultation - Debugging errors → suggest /investigate - Testing the app → suggest /qa - Code review before merge → suggest /review - Visual design audit → suggest /design-review - Ready to deploy / create PR → suggest /ship - Post-ship doc updates → suggest /document-release - Weekly retrospective → suggest /retro - Wanting a second opinion or adversarial code review → suggest /codex - Working with production or live systems → suggest /careful - Want to scope edits to one module/directory → suggest /freeze - Maximum safety mode (destructive warnings + edit restrictions) → suggest /guard - Removing edit restrictions → suggest /unfreeze - Upgrading gstack to latest version → suggest /gstack-upgrade If the user pushes back on skill suggestions ("stop suggesting things", "I don't need suggestions", "too aggressive"): 1. Stop suggesting for the rest of this session 2. Run: gstack-config set proactive false 3. Say: "Got it — I'll stop suggesting skills. Just tell me to be proactive again if you change your mind." If the user says "be proactive again" or "turn on suggestions": 1. Run: gstack-config set proactive true 2. Say: "Proactive suggestions are back on."
test/helpers/llm-judge.ts
View on GitHub/**
* Shared LLM-as-judge helpers for eval and E2E tests.
*
* Provides callJudge (generic JSON-from-LLM), judge (doc quality scorer),
* and outcomeJudge (planted-bug detection scorer).
*
* Requires: ANTHROPIC_API_KEY env var
*/
import Anthropic from '@anthropic-ai/sdk';
export interface JudgeScore {
clarity: number; // 1-5
completeness: number; // 1-5
actionability: number; // 1-5
reasoning: string;
}
export interface OutcomeJudgeResult {
detected: string[];
missed: string[];
false_positives: number;
detection_rate: number;
evidence_quality: number;
reasoning: string;
}
/**
* Call claude-sonnet-4-6 with a prompt, extract JSON response.
* Retries once on 429 rate limit errors.
*/
export async function callJudge<T>(prompt: string): Promise<T> {
const client = new Anthropic();
const makeRequest = () => client.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: 1024,
messages: [{ role: 'user', content: prompt }],
});
let response;
try {
response = await makeRequest();
} catch (err: any) {
if (err.status === 429) {
await new Promise(r => setTimeout(r, 1000));
response = await makeRequest();
} else {
throw err;
}
}
const text = response.content[0].type === 'text' ? response.content[0].text : '';
const jsonMatch = text.match(/\{[\s\S]*\}/);
if (!jsonMatch) throw new Error(`Judge returned non-JSON: ${text.slice(0, 200)}`);
return JSON.parse(jsonMatch[0]) as T;
}
/**
* Score documentation quality on clarity/completeness/actionability (1-5).
*/
export async function judge(section: string, content: string): Promise<JudgeScore> {
return callJudge<JudgeScore>(`You are evaluating documentation quality for an AI coding agent's CLI tool reference.
The agent reads this documentation to learn how to use a headless browser CLI. It needs to:
1. Understand what each command does
2. Know what arguments to pass
3. Know valid values for enum-like parameters
4. Construct correct command invocations without guessing
Rate the following ${section} on three dimensions (1-5 scale):
- **clarity** (1-5): Can an agent understand what each command/flag does from the description alone?
- **completeness** (1-5): Are arguments, valid values, and important behaviors documented? Would an agent need to guess anything?
- **actionability** (1-5): Can an agent construct correct command invocations from this reference alone?
Scoring guide:
- 5: Excellent — no ambiguity, all info present
- 4: Good — minor gaps an experienced agent could infer
- 3: Adequate — some guessing required
- 2: Poor — significant info missing
- 1: Unusable — agent would fail without external help
Respond with ONLY valid JSON in this exact format:
{"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"}
Here is the ${section} to evaluate:
${content}`);
}
/**
* Evaluate a QA report against planted-bug ground truth.
* Returns detection metrics for the planted bugs.
*/
export async function outcomeJudge(
groundTruth: any,
report: string,
): Promise<OutcomeJudgeResult> {
return callJudge<OutcomeJudgeResult>(`You are evaluating a QA testing report against known ground truth bugs.
GROUND TRUTH (${groundTruth.total_bugs} planted bugs):
${JSON.stringify(groundTruth.bugs, null, 2)}
QA REPORT (generated by an AI agent):
${report}
For each planted bug, determine if the report identified it. A bug counts as
"detected" if the report describes the same defect, even if the wording differs.
Use the detection_hint keywords as guidance.
Also count false positives: issues in the report that don't correspond to any
planted bug AND aren't legitimate issues with the page.
Respond with ONLY valid JSON:
{
"detected": ["bug-id-1", "bug-id-2"],
"missed": ["bug-id-3"],
"false_positives": 0,
"detection_rate": 2,
"evidence_quality": 4,
"reasoning": "brief explanation"
}
Rules:
- "detected" and "missed" arrays must only contain IDs from the ground truth: ${groundTruth.bugs.map((b: any) => b.id).join(', ')}
- detection_rate = length of detected array
- evidence_quality (1-5): Do detected bugs have screenshots, repro steps, or specific element references?
5 = excellent evidence for every bug, 1 = no evidence at all`);
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |