Skip to content

alexandervashurin/semaphore / .claude/skills/gstack

gstack

Flagged

Fast headless browser for QA testing and site dogfooding. Navigate any URL, interact with elements, verify page state, diff before/after actions, take annotated screenshots, check responsive layouts, test forms and uploads, handle dialogs, and assert element states. ~100ms per command. Use when you need to test a feature, verify a deployment, dogfood a user flow, or file a bug with evidence. gstack also includes development workflow skills. When you notice the user is at these stages, suggest the appropriate skill: - Brainstorming a new idea → suggest /office-hours - Reviewing a plan (strategy) → suggest /plan-ceo-review - Reviewing a plan (architecture) → suggest /plan-eng-review - Reviewing a plan (design) → suggest /plan-design-review - Creating a design system → suggest /design-consultation - Debugging errors → suggest /investigate - Testing the app → suggest /qa - Code review before merge → suggest /review - Visual design audit → suggest /design-review - Ready to deploy / create PR → suggest /ship - Post-ship doc updates → suggest /document-release - Weekly retrospective → suggest /retro - Wanting a second opinion or adversarial code review → suggest /codex - Working with production or live systems → suggest /careful - Want to scope edits to one module/directory → suggest /freeze - Maximum safety mode (destructive warnings + edit restrictions) → suggest /guard - Removing edit restrictions → suggest /unfreeze - Upgrading gstack to latest version → suggest /gstack-upgrade If the user pushes back on skill suggestions ("stop suggesting things", "I don't need suggestions", "too aggressive"): 1. Stop suggesting for the rest of this session 2. Run: gstack-config set proactive false 3. Say: "Got it — I'll stop suggesting skills. Just tell me to be proactive again if you change your mind." If the user says "be proactive again" or "turn on suggestions": 1. Run: gstack-config set proactive true 2. Say: "Proactive suggestions are back on."

FileKindSize
.env.exampleother171 B
.github/workflows/skill-docs.ymlconfig648 B
.gitignoreother137 B
AGENTS.mdmarkdown2.5 KB
ARCHITECTURE.mdmarkdown20.7 KB
BROWSER.mdmarkdown16.5 KB
CHANGELOG.mdmarkdown73.3 KB
CLAUDE.mdmarkdown12.8 KB
CONTRIBUTING.mdmarkdown15.1 KB
ETHOS.mdmarkdown5.5 KB
LICENSEother1.0 KB
README.mdmarkdown21.5 KB
SKILL.mdskill md27.2 KB
SKILL.md.tmplother8.4 KB
TODOS.mdmarkdown21.9 KB
VERSIONother8 B
bin/dev-setupother2.4 KB
bin/dev-teardownother1.6 KB
bin/gstack-analyticsother6.3 KB
bin/gstack-community-dashboardother4.7 KB
bin/gstack-configother1.1 KB
bin/gstack-diff-scopeother2.4 KB
bin/gstack-review-logother409 B
bin/gstack-review-readother526 B
bin/gstack-slugother478 B
bin/gstack-telemetry-logother6.7 KB
bin/gstack-telemetry-syncother4.4 KB
bin/gstack-update-checkother6.9 KB
conductor.jsonconfig87 B
docs/images/github-2013.pngbinary61.9 KB
docs/images/github-2026.pngbinary59.1 KB
docs/skills.mdmarkdown44.2 KB
package.jsonmanifest3.1 KB
scripts/analytics.tsscript5.3 KB
scripts/dev-skill.tsscript2.4 KB
scripts/eval-compare.tsscript3.0 KB
scripts/eval-list.tsscript3.5 KB
scripts/eval-select.tsscript3.1 KB
scripts/eval-summary.tsscript6.4 KB
scripts/eval-watch.tsscript5.4 KB
scripts/gen-skill-docs.tsscript89.5 KB
scripts/skill-check.tsscript5.8 KB
setupother8.4 KB
supabase/config.shscript454 B
supabase/functions/community-pulse/index.tsscript1.7 KB
supabase/functions/telemetry-ingest/index.tsscript4.1 KB
supabase/functions/update-check/index.tsscript1.1 KB
supabase/migrations/001_telemetry.sqlother3.2 KB
test/analytics.test.tsscript10.2 KB
test/codex-e2e.test.tsscript7.5 KB
test/fixtures/eval-baselines.jsonconfig388 B
test/fixtures/qa-eval-checkout-ground-truth.jsonconfig1.6 KB
test/fixtures/qa-eval-ground-truth.jsonconfig1.5 KB
test/fixtures/qa-eval-spa-ground-truth.jsonconfig1.6 KB
test/fixtures/review-eval-design-slop.cssother1.5 KB
test/fixtures/review-eval-design-slop.htmlother1.3 KB
test/fixtures/review-eval-enum-diff.rbscript986 B
test/fixtures/review-eval-enum.rbscript759 B
test/fixtures/review-eval-vuln.rbscript406 B
test/gemini-e2e.test.tsscript6.3 KB
test/gen-skill-docs.test.tsscript37.3 KB
test/helpers/codex-session-runner.tsscript8.9 KB
test/helpers/e2e-helpers.tsscript9.5 KB
test/helpers/eval-store.test.tsscript19.8 KB
test/helpers/eval-store.tsscript23.6 KB
test/helpers/gemini-session-runner.test.tsscript4.4 KB
test/helpers/gemini-session-runner.tsscript6.0 KB
test/helpers/llm-judge.tsscript4.2 KB
test/helpers/observability.test.tsscript9.9 KB
test/helpers/session-runner.test.tsscript3.7 KB
test/helpers/session-runner.tsscript11.6 KB
test/helpers/skill-parser.tsscript5.6 KB
test/helpers/touchfiles.tsscript11.0 KB
test/hook-scripts.test.tsscript13.6 KB
test/skill-e2e-browse.test.tsscript11.8 KB
test/skill-e2e-deploy.test.tsscript11.2 KB
test/skill-e2e-design.test.tsscript24.4 KB
test/skill-e2e-plan.test.tsscript19.7 KB
test/skill-e2e-qa-bugs.test.tsscript8.1 KB
test/skill-e2e-qa-workflow.test.tsscript15.4 KB
test/skill-e2e-review.test.tsscript23.2 KB
test/skill-e2e-workflow.test.tsscript23.2 KB
test/skill-llm-eval.test.tsscript33.6 KB
test/skill-parser.test.tsscript5.3 KB
test/skill-routing-e2e.test.tsscript26.3 KB
test/skill-validation.test.tsscript54.5 KB
test/telemetry.test.tsscript10.3 KB
test/touchfiles.test.tsscript10.1 KB

test/codex-e2e.test.ts

View on GitHub
/**
 * Codex CLI E2E tests — verify skills work when invoked by Codex.
 *
 * Spawns `codex exec` with skills installed in a temp HOME, parses JSONL
 * output, and validates structured results. Follows the same pattern as
 * skill-e2e.test.ts but adapted for Codex CLI.
 *
 * Prerequisites:
 * - `codex` binary installed (npm install -g @openai/codex)
 * - Codex authenticated via ~/.codex/ config (no OPENAI_API_KEY env var needed)
 * - EVALS=1 env var set (same gate as Claude E2E tests)
 *
 * Skips gracefully when prerequisites are not met.
 */

import { describe, test, expect, afterAll } from 'bun:test';
import { runCodexSkill, parseCodexJSONL, installSkillToTempHome } from './helpers/codex-session-runner';
import type { CodexResult } from './helpers/codex-session-runner';
import { EvalCollector } from './helpers/eval-store';
import type { EvalTestEntry } from './helpers/eval-store';
import { selectTests, detectBaseBranch, getChangedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES } from './helpers/touchfiles';
import * as fs from 'fs';
import * as path from 'path';
import * as os from 'os';

const ROOT = path.resolve(import.meta.dir, '..');

// --- Prerequisites check ---

const CODEX_AVAILABLE = (() => {
  try {
    const result = Bun.spawnSync(['which', 'codex']);
    return result.exitCode === 0;
  } catch { return false; }
})();

const evalsEnabled = !!process.env.EVALS;

// Skip all tests if codex is not available or EVALS is not set.
// Note: Codex uses its own auth from ~/.codex/ config — no OPENAI_API_KEY env var needed.
const SKIP = !CODEX_AVAILABLE || !evalsEnabled;

const describeCodex = SKIP ? describe.skip : describe;

// Log why we're skipping (helpful for debugging CI)
if (!evalsEnabled) {
  // Silent — same as Claude E2E tests, EVALS=1 required
} else if (!CODEX_AVAILABLE) {
  process.stderr.write('\nCodex E2E: SKIPPED — codex binary not found (install: npm i -g @openai/codex)\n');
}

// --- Diff-based test selection ---

// Codex E2E touchfiles — keyed by test name, same pattern as E2E_TOUCHFILES
const CODEX_E2E_TOUCHFILES: Record<string, string[]> = {
  'codex-discover-skill':    ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts'],
  'codex-review-findings':   ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts'],
};

let selectedTests: string[] | null = null; // null = run all

if (evalsEnabled && !process.env.EVALS_ALL) {
  const baseBranch = process.env.EVALS_BASE
    || detectBaseBranch(ROOT)
    || 'main';
  const changedFiles = getChangedFiles(baseBranch, ROOT);

  if (changedFiles.length > 0) {
    const selection = selectTests(changedFiles, CODEX_E2E_TOUCHFILES, GLOBAL_TOUCHFILES);
    selectedTests = selection.selected;
    process.stderr.write(`\nCodex E2E selection (${selection.reason}): ${selection.selected.length}/${Object.keys(CODEX_E2E_TOUCHFILES).length} tests\n`);
    if (selection.skipped.length > 0) {
      process.stderr.write(`  Skipped: ${selection.skipped.join(', ')}\n`);
    }
    process.stderr.write('\n');
  }
  // If changedFiles is empty (e.g., on main branch), selectedTests stays null -> run all
}

/** Skip an individual test if not selected by diff-based selection. */
function testIfSelected(testName: string, fn: () => Promise<void>, timeout: number) {
  const shouldRun = selectedTests === null || selectedTests.includes(testName);
  (shouldRun ? test.concurrent : test.skip)(testName, fn, timeout);
}

// --- Eval result collector ---

const evalCollector = evalsEnabled && !SKIP ? new EvalCollector('e2e-codex') : null;

/** DRY helper to record a Codex E2E test result into the eval collector. */
function recordCodexE2E(name: string, result: CodexResult, passed: boolean) {
  evalCollector?.addTest({
    name,
    suite: 'codex-e2e',
    tier: 'e2e',
    passed,
    duration_ms: result.durationMs,
    cost_usd: 0, // Codex doesn't report cost in the same way; tokens are tracked
    output: result.output?.slice(0, 2000),
    turns_used: result.toolCalls.length, // approximate: tool calls as turns
    exit_reason: result.exitCode === 0 ? 'success' : `exit_code_${result.exitCode}`,
  });
}

/** Print cost summary after a Codex E2E test. */
function logCodexCost(label: string, result: CodexResult) {
  const durationSec = Math.round(result.durationMs / 1000);
  console.log(`${label}: ${result.tokens} tokens, ${result.toolCalls.length} tool calls, ${durationSec}s`);
}

// Finalize eval results on exit
afterAll(async () => {
  if (evalCollector) {
    await evalCollector.finalize();
  }
});

// --- Tests ---

describeCodex('Codex E2E', () => {

  testIfSelected('codex-discover-skill', async () => {
    // Install gstack-review skill to a temp HOME and ask Codex to list skills
    const skillDir = path.join(ROOT, '.agents', 'skills', 'gstack-review');

    const result = await runCodexSkill({
      skillDir,
      prompt: 'List any skills or instructions you have available. Just list the names.',
      timeoutMs: 60_000,
      cwd: ROOT,
      skillName: 'gstack-review',
    });

    logCodexCost('codex-discover-skill', result);

    // Codex should have produced some output
    const passed = result.exitCode === 0 && result.output.length > 0;
    recordCodexE2E('codex-discover-skill', result, passed);

    expect(result.exitCode).toBe(0);
    expect(result.output.length).toBeGreaterThan(0);
    // The output should reference the skill name in some form
    const outputLower = result.output.toLowerCase();
    expect(
      outputLower.includes('review') || outputLower.includes('gstack') || outputLower.includes('skill'),
    ).toBe(true);
  }, 120_000);

  // Validates that Codex can invoke the gstack-review skill, run a diff-based
  // code review, and produce structured review output with findings/issues.
  // Accepts Codex timeout (exit 124/137) as non-failure since that's a CLI perf issue.
  testIfSelected('codex-review-findings', async () => {
    // Install gstack-review skill and ask Codex to review the current repo
    const skillDir = path.join(ROOT, '.agents', 'skills', 'gstack-review');

    const result = await runCodexSkill({
      skillDir,
      prompt: 'Run the gstack-review skill on this repository. Review the current branch diff and report your findings.',
      timeoutMs: 540_000,
      cwd: ROOT,
      skillName: 'gstack-review',
    });

    logCodexCost('codex-review-findings', result);

    // Should produce structured review-like output
    const output = result.output;

    // Codex may time out on large diffs — accept timeout as "not our fault"
    // exitCode 124 = killed by timeout, which is a Codex CLI performance issue
    if (result.exitCode === 124 || result.exitCode === 137) {
      console.warn(`codex-review-findings: Codex timed out (exit ${result.exitCode}) — skipping assertions`);
      recordCodexE2E('codex-review-findings', result, true); // don't fail the suite
      return;
    }

    const passed = result.exitCode === 0 && output.length > 50;
    recordCodexE2E('codex-review-findings', result, passed);

    expect(result.exitCode).toBe(0);
    expect(output.length).toBeGreaterThan(50);

    // Review output should contain some review-like content
    const outputLower = output.toLowerCase();
    const hasReviewContent =
      outputLower.includes('finding') ||
      outputLower.includes('issue') ||
      outputLower.includes('review') ||
      outputLower.includes('change') ||
      outputLower.includes('diff') ||
      outputLower.includes('clean') ||
      outputLower.includes('no issues') ||
      outputLower.includes('p1') ||
      outputLower.includes('p2');
    expect(hasReviewContent).toBe(true);
  }, 600_000);
});
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 465kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 465kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 465kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 192kUpdated Claude CodeCodexVerifiedInstall