notque/vexjoy-agent / skills/process/testing
testing
VerifiedTesting: TDD, E2E, test automation strategy (Vitest, coverage, CI), preferred patterns, test-value audits, verification, agent testing.
references/verify-verification-examples.md
View on GitHub# Verification Examples: Good vs Bad
This document shows real-world examples of good and bad verification practices to illustrate the defense-in-depth approach.
## Example 1: Python Bug Fix
### Insufficient Verification
**User**: Fix the login bug where passwords aren't being validated
**Assistant**: I've fixed the password validation in auth.py. The bug was that we weren't checking password length. Should be working now!
**Problems**:
- No tests run
- No code shown
- No verification of the fix
- Says "should be working" without evidence
- User has no confidence the fix works
---
### Thorough Verification
**User**: Fix the login bug where passwords aren't being validated
**Assistant**: I'll fix the password validation issue. Let me identify the problem and verify the fix.
**Step 1: Review the code**
```bash
$ cat -n /home/project/auth.py
15 def validate_password(password):
16 # TODO: Add validation
17 return True
```
**Issue found**: Password validation always returns True
**Step 2: Implement fix**
```python
def validate_password(password):
if len(password) < 8:
return False
if not any(c.isupper() for c in password):
return False
if not any(c.isdigit() for c in password):
return False
return True
```
**Step 3: Run tests**
```bash
$ pytest tests/test_auth.py -v
============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-7.4.0
collected 5 tests
tests/test_auth.py::test_password_too_short PASSED [ 20%]
tests/test_auth.py::test_password_no_uppercase PASSED [ 40%]
tests/test_auth.py::test_password_no_digit PASSED [ 60%]
tests/test_auth.py::test_password_valid PASSED [ 80%]
tests/test_auth.py::test_login_with_invalid_password PASSED [100%]
============================== 5 passed in 0.23s ==============================
```
**Step 4: Verify syntax**
```bash
$ python -m py_compile auth.py
$ echo $?
0
```
**Step 5: Check for unintended changes**
```bash
$ git diff auth.py
--- a/auth.py
+++ b/auth.py
@@ -14,7 +14,12 @@
def validate_password(password):
- # TODO: Add validation
- return True
+ if len(password) < 8:
+ return False
+ if not any(c.isupper() for c in password):
+ return False
+ if not any(c.isdigit() for c in password):
+ return False
+ return True
```
**Verification Complete**:
- ✅ Tests pass (5/5)
- ✅ Syntax valid
- ✅ Only intended changes
- ✅ Password validation now enforces: min 8 chars, uppercase, digit
Test if this addresses the issue.
**Why this is better**:
- Shows actual test output (5 specific tests)
- Verifies syntax
- Shows git diff to prove only intended changes
- User can see exactly what was fixed
- No ambiguous "should work" language
---
## Example 2: Go Refactoring
### Insufficient Verification
**User**: Refactor the user service to use the new database package
**Assistant**: Done! I've refactored user.go to use the new db package. All the imports are updated and it should compile fine.
**Problems**:
- No build verification
- No test execution
- "Should compile" is speculation
- No evidence provided
---
### Thorough Verification
**User**: Refactor the user service to use the new database package
**Assistant**: I'll refactor the user service to use the new database package and verify everything still works.
**Step 1: Review changes**
```bash
$ git diff --name-only
internal/user/service.go
internal/user/repository.go
```
**Step 2: Run tests**
```bash
$ go test ./internal/user/... -v
=== RUN TestUserService_Create
--- PASS: TestUserService_Create (0.01s)
=== RUN TestUserService_Get
--- PASS: TestUserService_Get (0.01s)
=== RUN TestUserService_Update
--- PASS: TestUserService_Update (0.01s)
=== RUN TestUserService_Delete
--- PASS: TestUserService_Delete (0.01s)
=== RUN TestUserRepository_FindByEmail
--- PASS: TestUserRepository_FindByEmail (0.02s)
PASS
ok github.com/example/internal/user 0.156s
```
**Step 3: Run race detector**
```bash
$ go test -race ./internal/user/...
PASS
ok github.com/example/internal/user 0.312s
```
**Step 4: Verify build**
```bash
$ go build ./...
$ echo $?
0
```
**Step 5: Check imports**
```bash
$ head -20 internal/user/service.go
package user
import (
"context"
"github.com/example/internal/db" // New import
"github.com/example/internal/models"
)
```
**Step 6: Verify no print statements**
```bash
$ grep -r "fmt.Println" internal/user/
$ echo $?
1
```
(Exit code 1 means no matches found - good!)
**Verification Complete**:
- ✅ Tests pass (5/5) with race detector
- ✅ Build succeeds
- ✅ New db package imported correctly
- ✅ No debug print statements
- ✅ Files reviewed: service.go, repository.go
Test if this addresses the issue.
---
## Example 3: JavaScript Feature Addition
### Insufficient Verification
**User**: Add a "Clear All" button to the dashboard
**Assistant**: I've added the Clear All button to Dashboard.jsx. It's wired up to call clearAllItems(). Everything looks good!
**Problems**:
- No build verification
- No tests run
- No visual confirmation
- "Looks good" is subjective
---
### Thorough Verification
**User**: Add a "Clear All" button to the dashboard
**Assistant**: I'll add the Clear All button and verify it works correctly.
**Step 1: Review changes**
```bash
$ git diff --name-only
src/components/Dashboard.jsx
src/components/Dashboard.test.jsx
```
**Step 2: Check syntax**
```bash
$ node -c src/components/Dashboard.jsx
$ echo $?
0
```
**Step 3: Run tests**
```bash
$ npm test -- Dashboard.test.jsx
PASS src/components/Dashboard.test.jsx
Dashboard Component
✓ renders dashboard with items (45 ms)
✓ displays Clear All button (12 ms)
✓ calls clearAllItems when button clicked (23 ms)
✓ disables button when no items (8 ms)
Test Suites: 1 passed, 1 total
Tests: 4 passed, 4 total
Snapshots: 0 total
Time: 1.234 s
```
**Step 4: Build verification**
```bash
$ npm run build
> app@1.0.0 build
> react-scripts build
Creating an optimized production build...
Compiled successfully.
File sizes after gzip:
52.3 kB build/static/js/main.a1b2c3d4.js
1.8 kB build/static/css/main.e5f6g7h8.css
The build folder is ready to be deployed.
```
**Step 5: Check for console.log**
```bash
$ grep -n "console.log" src/components/Dashboard.jsx
$ echo $?
1
```
(No console.log statements found)
**Step 6: Review actual code**
```jsx
// Dashboard.jsx excerpt
<button
className="btn btn-danger"
onClick={clearAllItems}
disabled={items.length === 0}
>
Clear All
</button>
```
**Verification Complete**:
- ✅ Syntax valid
- ✅ Tests pass (4 new tests)
- ✅ Build succeeds (52.3 kB bundle)
- ✅ No console.log statements
- ✅ Button properly wired to clearAllItems()
- ✅ Button disabled when no items
Test if this addresses the issue.
---
## Example 4: Database Migration
### Insufficient Verification
**User**: Add an index on the email column in the users table
**Assistant**: Index added! I created the migration file and the index should improve query performance.
**Problems**:
- Migration not tested
- No verification it actually runs
- "Should improve" is speculation
- Doesn't check if index was created
---
### Thorough Verification
**User**: Add an index on the email column in the users table
**Assistant**: I'll create the migration and verify it works correctly.
**Step 1: Create migration**
```bash
$ ls migrations/
20250129_add_email_index.sql
```
**Step 2: Review migration content**
```sql
-- migrations/20250129_add_email_index.sql
-- Add index on users.email for faster lookups
BEGIN;
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email
ON users(email);
COMMIT;
```
**Step 3: Test migration on test database**
```bash
$ psql test_db -f migrations/20250129_add_email_index.sql
BEGIN
CREATE INDEX
COMMIT
```
**Step 4: Verify index exists**
```bash
$ psql test_db -c "\d users"
Table "public.users"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
id | integer | | not null |
email | varchar | | |
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
"idx_users_email" btree (email) <--- NEW INDEX
```
**Step 5: Test query performance**
```bash
$ psql test_db -c "EXPLAIN SELECT * FROM users WHERE email = 'test@example.com';"
QUERY PLAN
---------------------------------------------------------------------------------
Index Scan using idx_users_email on users (cost=0.42..8.44 rows=1 width=100)
Index Cond: ((email)::text = 'test@example.com'::text)
```
(Using index scan - good!)
**Step 6: Test rollback**
```sql
-- migrations/20250129_add_email_index_down.sql
BEGIN;
DROP INDEX CONCURRENTLY IF EXISTS idx_users_email;
COMMIT;
```
```bash
$ psql test_db -f migrations/20250129_add_email_index_down.sql
BEGIN
DROP INDEX
COMMIT
$ psql test_db -c "\d users" | grep idx_users_email
$ echo $?
1
```
(Index successfully removed by rollback)
**Verification Complete**:
- ✅ Migration file created
- ✅ Migration runs successfully
- ✅ Index actually exists (verified with \d)
- ✅ Query uses index (verified with EXPLAIN)
- ✅ Rollback works correctly
- ✅ Uses CONCURRENTLY to avoid locks
Test if this addresses the issue.
---
## Example 5: Configuration Change
### Insufficient Verification
**User**: Update nginx config to add rate limiting
**Assistant**: Updated nginx.conf with rate limiting. The config should prevent abuse now.
**Problems**:
- Config not syntax-checked
- Not tested
- "Should prevent" is speculation
- No verification nginx accepts it
---
### Thorough Verification
**User**: Update nginx config to add rate limiting
**Assistant**: I'll update the nginx configuration and verify it's valid.
**Step 1: Review changes**
```bash
$ git diff nginx.conf
@@ -10,6 +10,9 @@ http {
include mime.types;
default_type application/octet-stream;
+ # Rate limiting
+ limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
+
server {
listen 80;
server_name example.com;
@@ -18,6 +21,8 @@ http {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
+
+ limit_req zone=api_limit burst=20 nodelay;
}
}
}
```
**Step 2: Syntax validation**
```bash
$ nginx -t -c /etc/nginx/nginx.conf
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```
**Step 3: Check rate limit zone**
```bash
$ grep -A 1 "limit_req_zone" nginx.conf
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
```
**Step 4: Dry-run reload**
```bash
$ nginx -s reload
2025/11/29 10:30:00 [notice] signal process started
```
(No errors - successful reload)
**Step 5: Test rate limiting**
```bash
$ for i in {1..15}; do curl -w "%{http_code}\n" -o /dev/null -s http://localhost/api/test; done
200
200
200
200
200
200
200
200
200
200
503
503
503
503
503
```
(First 10 requests succeed, next 5 rate-limited with 503)
**Verification Complete**:
- ✅ Syntax valid (nginx -t passed)
- ✅ Configuration reload successful
- ✅ Rate limit zone created (10m size, 10r/s limit)
- ✅ Rate limiting actually works (tested with curl)
- ✅ Burst handling configured (burst=20)
Test if this addresses the issue.
---
## Key Patterns in Good Verification
### 1. Multiple Independent Checks
Good verification uses multiple different validation methods:
- Syntax checkers (py_compile, node -c, gofmt)
- Tests (pytest, go test, npm test)
- Build verification (go build, npm run build)
- Runtime checks (start server, query database)
- Manual inspection (Read tool, git diff)
### 2. Show Actual Output
Never summarize - always show:
- Complete test results with test names
- Full build output
- Actual command output
- Exit codes
### 3. Evidence-Based Claims
Replace speculation with evidence:
- ❌ "Should work now"
- ✅ "Tests pass: [show output]"
- ❌ "Looks good"
- ✅ "Syntax valid: [show command]"
- ❌ "Index improves performance"
- ✅ "EXPLAIN shows index scan: [show output]"
### 4. Check for Unintended Changes
Always verify:
```bash
git diff # Review all changes
grep "console.log" # No debug code
grep "TODO" # No leftover TODOs
grep "password\|secret" # No credentials
```
### 5. Domain-Appropriate Verification
Different domains need different checks:
- **Python**: pytest + syntax + imports
- **Go**: tests + race detector + build + gofmt
- **JavaScript**: tests + syntax + build + bundle size
- **Database**: migration test + rollback test + EXPLAIN
- **Config**: syntax check + dry-run + actual reload
### 6. User-Friendly Format
Good verification provides:
- Clear step numbers
- Actual commands with $ prefix
- Complete output (not summaries)
- Checklist summary at end
- "Test if this addresses the issue" invitation
---
## Red Flags in Bad Verification
Watch for these warning signs:
1. **Speculation Language**:
- "Should work"
- "Should fix"
- "Should prevent"
- "Looks good"
- "Seems fine"
2. **Missing Evidence**:
- "Tests pass" (without showing output)
- "Build succeeds" (without showing command)
- "No errors" (without showing what was checked)
3. **Skipped Verification**:
- "Small change, no need to test"
- "Just config, won't break anything"
- "Only comments changed"
4. **Incomplete Checks**:
- Only syntax check, no tests
- Only read code, no execution
- Only manual review, no automation
5. **No User Confidence**:
- User must still test blind
- No evidence changes work
- No visibility into verification process
---
## Verification Mindset
**Bad Mindset**: "I changed the code, it should work"
**Good Mindset**: "I changed the code, let me prove it works through multiple independent verification layers"
**Defense-in-Depth Principle**:
- **Layer 1**: Syntax validation
- **Layer 2**: Unit tests
- **Layer 3**: Integration tests
- **Layer 4**: Build verification
- **Layer 5**: Manual code review
- **Layer 6**: Runtime verification
Each layer catches different types of errors. Never rely on just one layer.
---
## Final Checklist
Before saying "done":
- [ ] Tests run (output shown)
- [ ] Build successful (output shown)
- [ ] Syntax validated (command shown)
- [ ] Files reviewed (Read tool used)
- [ ] Diff checked (git diff shown)
- [ ] No debug code (grep verified)
- [ ] No speculation (only evidence-based statements)
- [ ] User invited to test ("Test if this addresses the issue")
**Remember**: The goal isn't to check boxes - it's to have genuine confidence the changes work correctly.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 470k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |