Skip to content

alexandervashurin/semaphore / .claude/skills/gstack

gstack

Flagged

Fast headless browser for QA testing and site dogfooding. Navigate any URL, interact with elements, verify page state, diff before/after actions, take annotated screenshots, check responsive layouts, test forms and uploads, handle dialogs, and assert element states. ~100ms per command. Use when you need to test a feature, verify a deployment, dogfood a user flow, or file a bug with evidence. gstack also includes development workflow skills. When you notice the user is at these stages, suggest the appropriate skill: - Brainstorming a new idea → suggest /office-hours - Reviewing a plan (strategy) → suggest /plan-ceo-review - Reviewing a plan (architecture) → suggest /plan-eng-review - Reviewing a plan (design) → suggest /plan-design-review - Creating a design system → suggest /design-consultation - Debugging errors → suggest /investigate - Testing the app → suggest /qa - Code review before merge → suggest /review - Visual design audit → suggest /design-review - Ready to deploy / create PR → suggest /ship - Post-ship doc updates → suggest /document-release - Weekly retrospective → suggest /retro - Wanting a second opinion or adversarial code review → suggest /codex - Working with production or live systems → suggest /careful - Want to scope edits to one module/directory → suggest /freeze - Maximum safety mode (destructive warnings + edit restrictions) → suggest /guard - Removing edit restrictions → suggest /unfreeze - Upgrading gstack to latest version → suggest /gstack-upgrade If the user pushes back on skill suggestions ("stop suggesting things", "I don't need suggestions", "too aggressive"): 1. Stop suggesting for the rest of this session 2. Run: gstack-config set proactive false 3. Say: "Got it — I'll stop suggesting skills. Just tell me to be proactive again if you change your mind." If the user says "be proactive again" or "turn on suggestions": 1. Run: gstack-config set proactive true 2. Say: "Proactive suggestions are back on."

FileKindSize
.env.exampleother171 B
.github/workflows/skill-docs.ymlconfig648 B
.gitignoreother137 B
AGENTS.mdmarkdown2.5 KB
ARCHITECTURE.mdmarkdown20.7 KB
BROWSER.mdmarkdown16.5 KB
CHANGELOG.mdmarkdown73.3 KB
CLAUDE.mdmarkdown12.8 KB
CONTRIBUTING.mdmarkdown15.1 KB
ETHOS.mdmarkdown5.5 KB
LICENSEother1.0 KB
README.mdmarkdown21.5 KB
SKILL.mdskill md27.2 KB
SKILL.md.tmplother8.4 KB
TODOS.mdmarkdown21.9 KB
VERSIONother8 B
bin/dev-setupother2.4 KB
bin/dev-teardownother1.6 KB
bin/gstack-analyticsother6.3 KB
bin/gstack-community-dashboardother4.7 KB
bin/gstack-configother1.1 KB
bin/gstack-diff-scopeother2.4 KB
bin/gstack-review-logother409 B
bin/gstack-review-readother526 B
bin/gstack-slugother478 B
bin/gstack-telemetry-logother6.7 KB
bin/gstack-telemetry-syncother4.4 KB
bin/gstack-update-checkother6.9 KB
conductor.jsonconfig87 B
docs/images/github-2013.pngbinary61.9 KB
docs/images/github-2026.pngbinary59.1 KB
docs/skills.mdmarkdown44.2 KB
package.jsonmanifest3.1 KB
scripts/analytics.tsscript5.3 KB
scripts/dev-skill.tsscript2.4 KB
scripts/eval-compare.tsscript3.0 KB
scripts/eval-list.tsscript3.5 KB
scripts/eval-select.tsscript3.1 KB
scripts/eval-summary.tsscript6.4 KB
scripts/eval-watch.tsscript5.4 KB
scripts/gen-skill-docs.tsscript89.5 KB
scripts/skill-check.tsscript5.8 KB
setupother8.4 KB
supabase/config.shscript454 B
supabase/functions/community-pulse/index.tsscript1.7 KB
supabase/functions/telemetry-ingest/index.tsscript4.1 KB
supabase/functions/update-check/index.tsscript1.1 KB
supabase/migrations/001_telemetry.sqlother3.2 KB
test/analytics.test.tsscript10.2 KB
test/codex-e2e.test.tsscript7.5 KB
test/fixtures/eval-baselines.jsonconfig388 B
test/fixtures/qa-eval-checkout-ground-truth.jsonconfig1.6 KB
test/fixtures/qa-eval-ground-truth.jsonconfig1.5 KB
test/fixtures/qa-eval-spa-ground-truth.jsonconfig1.6 KB
test/fixtures/review-eval-design-slop.cssother1.5 KB
test/fixtures/review-eval-design-slop.htmlother1.3 KB
test/fixtures/review-eval-enum-diff.rbscript986 B
test/fixtures/review-eval-enum.rbscript759 B
test/fixtures/review-eval-vuln.rbscript406 B
test/gemini-e2e.test.tsscript6.3 KB
test/gen-skill-docs.test.tsscript37.3 KB
test/helpers/codex-session-runner.tsscript8.9 KB
test/helpers/e2e-helpers.tsscript9.5 KB
test/helpers/eval-store.test.tsscript19.8 KB
test/helpers/eval-store.tsscript23.6 KB
test/helpers/gemini-session-runner.test.tsscript4.4 KB
test/helpers/gemini-session-runner.tsscript6.0 KB
test/helpers/llm-judge.tsscript4.2 KB
test/helpers/observability.test.tsscript9.9 KB
test/helpers/session-runner.test.tsscript3.7 KB
test/helpers/session-runner.tsscript11.6 KB
test/helpers/skill-parser.tsscript5.6 KB
test/helpers/touchfiles.tsscript11.0 KB
test/hook-scripts.test.tsscript13.6 KB
test/skill-e2e-browse.test.tsscript11.8 KB
test/skill-e2e-deploy.test.tsscript11.2 KB
test/skill-e2e-design.test.tsscript24.4 KB
test/skill-e2e-plan.test.tsscript19.7 KB
test/skill-e2e-qa-bugs.test.tsscript8.1 KB
test/skill-e2e-qa-workflow.test.tsscript15.4 KB
test/skill-e2e-review.test.tsscript23.2 KB
test/skill-e2e-workflow.test.tsscript23.2 KB
test/skill-llm-eval.test.tsscript33.6 KB
test/skill-parser.test.tsscript5.3 KB
test/skill-routing-e2e.test.tsscript26.3 KB
test/skill-validation.test.tsscript54.5 KB
test/telemetry.test.tsscript10.3 KB
test/touchfiles.test.tsscript10.1 KB

test/helpers/e2e-helpers.ts

View on GitHub
/**
 * Shared helpers for E2E test files.
 *
 * Extracted from the monolithic skill-e2e.test.ts to support splitting
 * tests across multiple files by category.
 */

import { describe, test, afterAll } from 'bun:test';
import type { SkillTestResult } from './session-runner';
import { EvalCollector, judgePassed } from './eval-store';
import type { EvalTestEntry } from './eval-store';
import { selectTests, detectBaseBranch, getChangedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES } from './touchfiles';
import { spawnSync } from 'child_process';
import * as fs from 'fs';
import * as path from 'path';
import * as os from 'os';

export const ROOT = path.resolve(import.meta.dir, '..', '..');

// Skip unless EVALS=1. Session runner strips CLAUDE* env vars to avoid nested session issues.
//
// BLAME PROTOCOL: When an eval fails, do NOT claim "pre-existing" or "not related
// to our changes" without proof. Run the same eval on main to verify. These tests
// have invisible couplings — preamble text, SKILL.md content, and timing all affect
// agent behavior. See CLAUDE.md "E2E eval failure blame protocol" for details.
export const evalsEnabled = !!process.env.EVALS;

// --- Diff-based test selection ---
// When EVALS_ALL is not set, only run tests whose touchfiles were modified.
// Set EVALS_ALL=1 to force all tests. Set EVALS_BASE to override base branch.
export let selectedTests: string[] | null = null; // null = run all

// EVALS_FAST: skip the 8 slowest tests (all Opus quality tests) for quick feedback
const FAST_EXCLUDED_TESTS = [
  'plan-ceo-review-selective', 'plan-ceo-review', 'retro', 'retro-base-branch',
  'design-consultation-core', 'design-consultation-existing',
  'qa-fix-loop', 'design-review-fix',
];

if (evalsEnabled && !process.env.EVALS_ALL) {
  const baseBranch = process.env.EVALS_BASE
    || detectBaseBranch(ROOT)
    || 'main';
  const changedFiles = getChangedFiles(baseBranch, ROOT);

  if (changedFiles.length > 0) {
    const selection = selectTests(changedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES);
    selectedTests = selection.selected;
    process.stderr.write(`\nE2E selection (${selection.reason}): ${selection.selected.length}/${Object.keys(E2E_TOUCHFILES).length} tests\n`);
    if (selection.skipped.length > 0) {
      process.stderr.write(`  Skipped: ${selection.skipped.join(', ')}\n`);
    }
    process.stderr.write('\n');
  }
  // If changedFiles is empty (e.g., on main branch), selectedTests stays null → run all
}

// Apply EVALS_FAST filter after diff-based selection
if (evalsEnabled && process.env.EVALS_FAST) {
  if (selectedTests === null) {
    // Run all minus excluded
    selectedTests = Object.keys(E2E_TOUCHFILES).filter(t => !FAST_EXCLUDED_TESTS.includes(t));
  } else {
    selectedTests = selectedTests.filter(t => !FAST_EXCLUDED_TESTS.includes(t));
  }
  process.stderr.write(`EVALS_FAST: excluded ${FAST_EXCLUDED_TESTS.length} slow tests, running ${selectedTests.length}\n\n`);
}

export const describeE2E = evalsEnabled ? describe : describe.skip;

/** Wrap a describe block to skip entirely if none of its tests are selected. */
export function describeIfSelected(name: string, testNames: string[], fn: () => void) {
  const anySelected = selectedTests === null || testNames.some(t => selectedTests!.includes(t));
  (anySelected ? describeE2E : describe.skip)(name, fn);
}

// Unique run ID for this E2E session — used for heartbeat + per-run log directory
export const runId = new Date().toISOString().replace(/[:.]/g, '').replace('T', '-').slice(0, 15);

export const browseBin = path.resolve(ROOT, 'browse', 'dist', 'browse');

// Check if Anthropic API key is available (needed for outcome evals)
export const hasApiKey = !!process.env.ANTHROPIC_API_KEY;

/**
 * Copy a directory tree recursively (files only, follows structure).
 */
export function copyDirSync(src: string, dest: string) {
  fs.mkdirSync(dest, { recursive: true });
  for (const entry of fs.readdirSync(src, { withFileTypes: true })) {
    const srcPath = path.join(src, entry.name);
    const destPath = path.join(dest, entry.name);
    if (entry.isDirectory()) {
      copyDirSync(srcPath, destPath);
    } else {
      fs.copyFileSync(srcPath, destPath);
    }
  }
}

/**
 * Set up browse shims (binary symlink, find-browse, remote-slug) in a tmpDir.
 */
export function setupBrowseShims(dir: string) {
  // Symlink browse binary
  const binDir = path.join(dir, 'browse', 'dist');
  fs.mkdirSync(binDir, { recursive: true });
  if (fs.existsSync(browseBin)) {
    fs.symlinkSync(browseBin, path.join(binDir, 'browse'));
  }

  // find-browse shim
  const findBrowseDir = path.join(dir, 'browse', 'bin');
  fs.mkdirSync(findBrowseDir, { recursive: true });
  fs.writeFileSync(
    path.join(findBrowseDir, 'find-browse'),
    `#!/bin/bash\necho "${browseBin}"\n`,
    { mode: 0o755 },
  );

  // remote-slug shim (returns test-project)
  fs.writeFileSync(
    path.join(findBrowseDir, 'remote-slug'),
    `#!/bin/bash\necho "test-project"\n`,
    { mode: 0o755 },
  );
}

/**
 * Print cost summary after an E2E test.
 */
export function logCost(label: string, result: { costEstimate: { turnsUsed: number; estimatedTokens: number; estimatedCost: number }; duration: number }) {
  const { turnsUsed, estimatedTokens, estimatedCost } = result.costEstimate;
  const durationSec = Math.round(result.duration / 1000);
  console.log(`${label}: $${estimatedCost.toFixed(2)} (${turnsUsed} turns, ${(estimatedTokens / 1000).toFixed(1)}k tokens, ${durationSec}s)`);
}

/**
 * Dump diagnostic info on planted-bug outcome failure (decision 1C).
 */
export function dumpOutcomeDiagnostic(dir: string, label: string, report: string, judgeResult: any) {
  try {
    const transcriptDir = path.join(dir, '.gstack', 'test-transcripts');
    fs.mkdirSync(transcriptDir, { recursive: true });
    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
    fs.writeFileSync(
      path.join(transcriptDir, `${label}-outcome-${timestamp}.json`),
      JSON.stringify({ label, report, judgeResult }, null, 2),
    );
  } catch { /* non-fatal */ }
}

/**
 * Create an EvalCollector for a specific suite. Returns null if evals are not enabled.
 */
export function createEvalCollector(suite: string): EvalCollector | null {
  return evalsEnabled ? new EvalCollector(suite) : null;
}

/** DRY helper to record an E2E test result into the eval collector. */
export function recordE2E(
  evalCollector: EvalCollector | null,
  name: string,
  suite: string,
  result: SkillTestResult,
  extra?: Partial<EvalTestEntry>,
) {
  // Derive last tool call from transcript for machine-readable diagnostics
  const lastTool = result.toolCalls.length > 0
    ? `${result.toolCalls[result.toolCalls.length - 1].tool}(${JSON.stringify(result.toolCalls[result.toolCalls.length - 1].input).slice(0, 60)})`
    : undefined;

  evalCollector?.addTest({
    name, suite, tier: 'e2e',
    passed: result.exitReason === 'success' && result.browseErrors.length === 0,
    duration_ms: result.duration,
    cost_usd: result.costEstimate.estimatedCost,
    transcript: result.transcript,
    output: result.output?.slice(0, 2000),
    turns_used: result.costEstimate.turnsUsed,
    browse_errors: result.browseErrors,
    exit_reason: result.exitReason,
    timeout_at_turn: result.exitReason === 'timeout' ? result.costEstimate.turnsUsed : undefined,
    last_tool_call: lastTool,
    model: result.model,
    first_response_ms: result.firstResponseMs,
    max_inter_turn_ms: result.maxInterTurnMs,
    ...extra,
  });
}

/** Finalize an eval collector (write results). */
export async function finalizeEvalCollector(evalCollector: EvalCollector | null) {
  if (evalCollector) {
    try {
      await evalCollector.finalize();
    } catch (err) {
      console.error('Failed to save eval results:', err);
    }
  }
}

// Pre-seed preamble state files so E2E tests don't waste turns on lake intro + telemetry prompts.
// These are one-time interactive prompts that burn 3-7 turns per test if not pre-seeded.
if (evalsEnabled) {
  const gstackDir = path.join(os.homedir(), '.gstack');
  fs.mkdirSync(gstackDir, { recursive: true });
  for (const f of ['.completeness-intro-seen', '.telemetry-prompted']) {
    const p = path.join(gstackDir, f);
    if (!fs.existsSync(p)) fs.writeFileSync(p, '');
  }
}

// Fail fast if Anthropic API is unreachable — don't burn through tests getting ConnectionRefused
if (evalsEnabled) {
  const check = spawnSync('sh', ['-c', 'echo "ping" | claude -p --max-turns 1 --output-format stream-json --verbose --dangerously-skip-permissions'], {
    stdio: 'pipe', timeout: 30_000,
  });
  const output = check.stdout?.toString() || '';
  if (output.includes('ConnectionRefused') || output.includes('Unable to connect')) {
    throw new Error('Anthropic API unreachable — aborting E2E suite. Fix connectivity and retry.');
  }
}

/** Skip an individual test if not selected (for multi-test describe blocks). */
export function testIfSelected(testName: string, fn: () => Promise<void>, timeout: number) {
  const shouldRun = selectedTests === null || selectedTests.includes(testName);
  (shouldRun ? test : test.skip)(testName, fn, timeout);
}

/** Concurrent version — runs in parallel with other concurrent tests within the same describe block. */
export function testConcurrentIfSelected(testName: string, fn: () => Promise<void>, timeout: number) {
  const shouldRun = selectedTests === null || selectedTests.includes(testName);
  (shouldRun ? test.concurrent : test.skip)(testName, fn, timeout);
}

export { judgePassed } from './eval-store';
export { EvalCollector } from './eval-store';
export type { EvalTestEntry } from './eval-store';
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall