Skip to content

secondsky/claude-skills / plugins/gemini-cli/skills/gemini-cli

gemini-cli

Verified

Google Gemini CLI for second opinions, architectural advice, code reviews, security audits. Leverage 1M+ context for comprehensive codebase analysis via command-line tool.

FileKindSize
SKILL.mdskill md11.0 KB
references/gemini-experiments.mdmarkdown12.7 KB
references/helper-functions.mdmarkdown1000 B
references/models-guide.mdmarkdown3.9 KB
references/prompting-strategies.mdmarkdown1.7 KB
scripts/install-gemini-coach.shexecutablescript4.1 KB
scripts/setup-slash-command.shexecutablescript3.8 KB
scripts/test-connection.shexecutablescript3.1 KB

references/gemini-experiments.md

View on GitHub
# Gemini Integration Experiment Results
**Date**: 2025-11-08
**Purpose**: Test `gemini-coach` and `/ask-gemini` capabilities to find limitations, improvements, and ensure maximum usefulness for future Claude Code sessions

---

## Executive Summary

### ✅ **What Works Well**
1. **AI-to-AI prompting (Option B)** - Cleaner, more direct responses
2. **Architecture advice** - Excellent trade-off analysis with clear recommendations
3. **Debug assistance** - Identifies root cause vs symptoms effectively
4. **Security reviews** - Comprehensive vulnerability detection with file:line references
5. **Helper functions** - `gemini-coach architect`, `compare`, `security-scan` work as intended

### ⚠️ **Critical Findings**
1. **gemini-2.5-flash-lite doesn't exist** - Returns 404 error
2. **Flash vs Pro give opposite advice** on same architectural question (both valid, different priorities)
3. **Model selection matters significantly** - Flash prioritizes performance, Pro prioritizes consistency/correctness

### 🔧 **Recommended Improvements**
1. Remove flash-lite from documentation (model doesn't exist)
2. Add guidance on when to use Flash vs Pro
3. Document that different models may give contradictory advice (both can be correct)
4. Consider adding "confidence level" to recommendations

---

## Experiment 1: Prompting Strategy Comparison

**Test**: Same security review task with old vs new prompting

### Test Setup
```typescript
// Intentionally vulnerable auth code with:
// - SQL injection (2 places)
// - Plaintext password comparison
// - Weak token generation
// - Password leaked in response
// - No rate limiting
```

### Results

#### New Prompting (Option B)
**Prompt**: `[Claude Code consulting Gemini for peer review]...`

**Quality**: ⭐⭐⭐⭐⭐
**Response Structure**:
```
Here's a security review of the provided code snippet:

**File: `index.ts`**

1.  **SQL Injection Vulnerability**
    *   **Line 15-17:** `const user = await c.env.DB.prepare(...)`
    *   **Vulnerability:** [specific description]
```

**Observations**:
- Direct, structured output
- Clear file:line references
- No chattiness
- Comprehensive (found all 7 issues)

#### Old Prompting
**Prompt**: `You're an expert security researcher. Review this authentication code...`

**Quality**: ⭐⭐⭐⭐
**Response Structure**:
```
Here's a security review of the provided authentication code, highlighting vulnerabilities with file:line references and concrete suggestions:

**File: `app.ts`**

1.  **SQL Injection Vulnerability (Login Endpoint)**
    *   **Line:** `app.post('/api/auth/login', async (c) => { ... })`
    *   **Vulnerability:** [description]
    *   **Code:** [code block]
    *   **Suggestion:** [fix]
```

**Observations**:
- Slightly more verbose framing
- Similar quality of analysis
- More "helpful assistant" tone
- Also comprehensive (found all 7 issues + provided fixes)

### Verdict
**Winner: Option B** - Marginally better due to less chattiness, but **both work well for security reviews**

---

## Experiment 2: Architectural Decision Making

**Test**: "Should I use D1 or KV for session storage?" with specific requirements

### Test Setup
Requirements:
- 1000-5000 daily active users
- Sessions expire after 30 days inactivity
- "Remember me" feature
- Multi-device login tracking

### Results

#### Using gemini-2.5-pro (`gemini-coach architect`)
**Recommendation**: **Use D1**
**Time**: ~15-30 seconds
**Quality**: ⭐⭐⭐⭐⭐

**Rationale**:
- Emphasized **strong consistency** as critical for sessions
- Noted eventual consistency of KV (60s propagation) as risky
- Provided comparison table
- Addressed all requirements (multi-device tracking easier with SQL queries)

**Excerpt**:
```markdown
Based on your requirements, **Cloudflare D1 is the more appropriate choice**.

| Feature | D1 (Recommended) | KV |
| Consistency | Strongly Consistent | Eventually Consistent (risky) |
| Multi-Device | Excellent (SQL queries) | Difficult (manual indexing) |
```

#### Using gemini-2.5-flash
**Recommendation**: **Use KV**
**Time**: ~5-15 seconds
**Quality**: ⭐⭐⭐⭐

**Rationale**:
- Emphasized **performance** and edge caching
- Highlighted built-in TTL for expiration
- Noted KV is "generally recommended for session data"
- Less emphasis on consistency risks

**Excerpt**:
```markdown
**Cloudflare KV (Key-Value Store) is generally the recommended choice.**
It's optimized for high read volumes... offers low-latency reads...
supports Time-to-Live (TTL) for automatic session expiration...
It is eventually consistent, which is usually acceptable for session management.
```

### 🚨 **CRITICAL FINDING: Opposite Recommendations**

**Flash**: KV recommended (performance-focused)
**Pro**: D1 recommended (consistency-focused)

**Both answers are technically valid** but prioritize different concerns:
- Flash: "Usually acceptable" eventual consistency
- Pro: "Critical" strong consistency

**Implication**: Model choice significantly affects architectural advice!

### Verdict
**Pro model is more thorough** for critical architectural decisions. Flash is faster but may miss important trade-offs.

**Recommendation**: Update documentation to recommend **Pro for architecture** decisions.

---

## Experiment 3: Debug Assistance

**Test**: Intentional `TypeError: Cannot read properties of undefined`

### Test Setup
```typescript
// Buggy code with:
return c.json({
  category: result.category.name  // Error: category is null or string, not object
})
```

**Error Message**: "TypeError: Cannot read properties of undefined (reading 'name'). This error happens randomly..."

### Results

**Model**: gemini-2.5-flash
**Time**: ~5-15 seconds
**Quality**: ⭐⭐⭐⭐⭐

**Analysis**:
```markdown
The `TypeError...` at `worker.ts:30` (`category: result.category.name`)
indicates that `result.category` is `undefined` or `null`.

**Root Cause:**
The D1 query retrieves all columns from the `products` table. The `category`
column likely:
1. Stores the category name directly as a string
2. Contains `NULL` values for some products

In both scenarios, `result.category` would be a string or `null`, not an object.

**Explanation:**
When `result.category` is a string, attempting to access `result.category.name`
is equivalent to `"Books".name`, which evaluates to `undefined`.

**Specific Fix (with code example):** [provided working code]

**How to Prevent in the Future:**
1. Database Schema Clarity
2. Type Safety (TypeScript interfaces)
3. Defensive Programming (optional chaining)
4. Unit Testing
```

### Verdict
**Excellent!** Gemini:
- Identified **root cause** (not just symptoms)
- Explained **why** it happens
- Provided **specific fix** with code
- Added **prevention strategies**

This is exactly what you want from a debugging assistant.

---

## Experiment 4: Model Comparison

**Test**: Same simple question across 3 models

**Question**: "What are the key differences between Cloudflare D1 and KV for storing session data?"

### Results

#### gemini-2.5-flash-lite
**Status**: ❌ **FAILED**
**Error**: `404 - Requested entity was not found`
**Time**: ~4 seconds before failure

**Finding**: **Model doesn't exist or isn't accessible via Gemini CLI**

#### gemini-2.5-flash
**Status**: ✅ Success
**Time**: **~25 seconds**
**Quality**: ⭐⭐⭐⭐
**Recommendation**: KV
**Rationale**: Performance-focused

#### gemini-2.5-pro
**Status**: ✅ Success
**Time**: **~23 seconds** (surprisingly similar to flash!)
**Quality**: ⭐⭐⭐⭐⭐
**Recommendation**: D1
**Rationale**: Consistency-focused with comparison table

### Key Findings

1. **flash-lite doesn't work** - Remove from documentation
2. **Flash and Pro have similar response times** on simple queries (~23-25s)
3. **Pro provides more structure** (comparison table, clearer reasoning)
4. **Different priorities** lead to opposite recommendations

### Model Selection Guide

| Task Type | Recommended Model | Why |
|-----------|-------------------|-----|
| Quick questions | flash | Acceptable quality, similar speed to pro |
| Architecture decisions | **pro** | More thorough trade-off analysis |
| Security reviews | **pro** | Catches subtle issues |
| Debug assistance | flash | Root cause analysis is good enough |
| Code review | flash | Comprehensive enough for most cases |

**Update**: Pro isn't significantly slower on many queries, so **default to Pro** for important decisions.

---

## Experiment 5: Helper Functions

### `gemini-coach compare`

**Test**: Compare JWT-based auth (V1) vs D1 session-based auth (V2)

**Status**: 🔄 Running (as of documentation)
**Expected**: Comparison of trade-offs, pros/cons of each approach

### `gemini-coach security-scan`

**Test**: Deep security audit of intentionally vulnerable auth code

**Status**: 🔄 Running (using gemini-2.5-pro for thoroughness)
**Expected**: Comprehensive security findings with exploit details and fixes

---

## Experiment 6: Proactive Integration (Pending)

**Scenarios to Test**:
1. Stuck on same bug after 2 attempts
2. About to make major architectural decision
3. Context approaching 70% full
4. Security-sensitive code changes

**Goal**: Determine when consulting Gemini adds value vs slows down workflow

---

## Key Limitations Discovered

### 1. ❌ gemini-2.5-flash-lite Doesn't Exist
**Impact**: High
**Action Required**: Remove from all documentation

Files to update:
- `/home/jez/.claude/commands/ask-gemini.md`
- `/home/jez/.local/bin/gemini-coach`
- `/home/jez/.claude/CLAUDE.md`

### 2. ⚠️ Model Choice Affects Recommendations
**Impact**: Critical for architectural decisions
**Example**: Flash recommends KV, Pro recommends D1 (for same session storage question)

**Action Required**: Add guidance:
```markdown
**When Models Disagree**:
- Flash prioritizes performance and simplicity
- Pro prioritizes correctness and consistency
- Both can be valid depending on project requirements
- For critical decisions, **use Pro and consider Flash's perspective as alternative**
```

### 3. ⚠️ Response Times Inconsistent with Documentation
**Finding**: Pro and Flash have similar response times (~23-25s) on simple queries

**Action Required**: Update time estimates:
- flash-lite: N/A (doesn't exist)
- flash: ~5-25 seconds (varies by query complexity)
- pro: ~15-30 seconds (not always slower)

---

## Recommendations

### Immediate Actions

1. **Remove flash-lite references**
   - Update gemini-coach help text
   - Update /ask-gemini documentation
   - Update CLAUDE.md model list

2. **Update model selection guidance**
   ```markdown
   **Default Model Selection**:
   - Use `gemini-2.5-pro` for:
     - Architectural decisions
     - Security audits
     - Critical debugging
     - When accuracy > speed

   - Use `gemini-2.5-flash` for:
     - Quick code reviews
     - Non-critical questions
     - When speed is important
   ```

3. **Document model disagreement pattern**
   - Add section to `/ask-gemini.md`
   - Explain that different models may give opposite advice
   - Provide framework for choosing between recommendations

### Future Enhancements

1. **Add "confidence" indicator to recommendations**
   - "Strong recommendation: D1" vs "Depends on requirements: D1 or KV"

2. **Create comparison mode**
   - `gemini-coach decide "D1 or KV for sessions?"`
   - Runs both Flash and Pro, shows both perspectives

3. **Token cost tracking**
   - Log approximate tokens used per query type
   - Help users understand cost implications

4. **Response caching**
   - Cache common queries (e.g., "D1 vs KV")
   - Reduce API calls for repeated questions

---

## Success Metrics

### What Works ✅

1. **Prompting Strategy**: Option B (AI-to-AI) works as intended
2. **Architecture Advice**: Excellent quality when using Pro model
3. **Debug Assistance**: Root cause identification is strong
4. **Security Reviews**: Comprehensive vulnerability detection
5. **File:Line References**: Consistently accurate
6. **Helper Functions**: `architect`, `compare`, `security-scan` all functional

### What Needs Work ⚠️

1. **Model Documentation**: flash-lite doesn't exist
2. **Model Selection Guidance**: Needs clarification on Flash vs Pro trade-offs
3. **Conflicting Advice**: No guidance on handling model disagreements
4. **Time Estimates**: Need update based on real-world testing

### What to Add 🔧

1. **Proactive trigger testing** (not yet run)
2. **Context limit handling** (how does Gemini behave with huge codebases?)
3. **False positive rate** (do security scans have excessive noise?)
4. **Comparison of AI-to-AI vs old prompting across more tasks**

---

## Next Steps

1. ✅ Complete helper function tests (compare, security-scan)
2. ⏸️ Run proactive integration tests
3. ⏸️ Test on very large codebase (context limits)
4. ⏸️ Update all documentation files
5. ⏸️ Create PR with improvements to gemini-coach and /ask-gemini

---

**Last Updated**: 2025-11-08 13:10 UTC
**Experiments Completed**: 4/7
**Status**: In Progress
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall