Skip to content

putervision/behavior-mcp / .agents/skills/vision-memory-mcp

vision-memory-mcp

Verified

Teaches the agent to use the Visual Memory MCP server to cache webpage and application screenshots, matching layout states and avoiding redundant LLM vision calls.

FileKindSize
SKILL.mdskill md5.4 KB
---
name: vision-memory-mcp
description: Teaches the agent to use the Visual Memory MCP server to cache webpage and application screenshots, matching layout states and avoiding redundant LLM vision calls.
---

# Visual Memory (vision-memory-mcp)

This project utilizes `vision-memory-mcp` to cache visual states, record layout transitions, and avoid repetitive LLM vision calls.

### 1. Priority Order & Checklist
Whenever you capture a screenshot, examine a webpage, or need to verify a visual state, you MUST run this sequence:
1. **Orient**: Call `get_session_context` to load active transitions and recent visual states.
2. **Search (Optional)**: Call `recall_memory` to search past visual states by text query or image query.
3. **Ingest/Verify**: Call `analyze_screenshot` with the base64 screenshot before calling any vision LLMs.
4. **Cache Hit**: If `is_known` is `true`, read the returned `description` and do NOT call your vision LLM.
5. **Cache Miss**: If `is_known` is `false`, inspect the image with your vision model, summarize the layout, and register it back by calling `analyze_screenshot` with both the `screenshot` and `description` parameters.
6. **Log Transitions**: Right after taking any UI action (click, type, navigate, scroll), call `record_outcome` to build the navigation graph.
7. **Snapshotting**: Call `manage_snapshot` (`action: "save"`) when reaching milestones, and `manage_snapshot` (`action: "diff"`) to check for visual regressions.

### 2. Complete Tool Reference

| Tool Name | Key Inputs | Description |
|-----------|------------|-------------|
| `analyze_screenshot` | `screenshot`? (base64), `file_path`?, `description`?, `items`? | Main ingestion (single or batch) and visual state retrieval tool. |
| `recall_memory` | `query`?, `screenshot`?, `file_path`?, `strategy`?, `limit`? | Search visual memory by text query or image query (read-only). |
| `record_outcome` | `from_state_id`, `to_state_id`?, `action`, `action_type`? ('blocker' \| 'click' \| etc.) | Record UI action transitions or log visual blockers for state-memory. |
| `get_navigation_paths` | `from_state_id`?, `to_state_id`?, `to_description`?, `max_hops`? | Find historical path or instructions between states. |
| `predict_next_action` | `current_state_id`, `goal_description`?, `goal_state_id`? | Predict best next UI action and grounded element handles (`target_selector`, `target_coords`). |
| `compare_states` | `state_a_id` & `state_b_id` OR `video_a_id` & `video_b_id` | Compare two states visually (`has_layout_change`) or compare video runs. |
| `get_session_context` | `include_recent`?, `include_frequent`? | Get recent/frequent states, transition graphs, disk stats, cache metrics, and version info. |
| `manage_snapshot` | `action` ('save' \| 'diff' \| 'export' \| 'restore'), `name`?, `archive_json`? | Unified snapshot management for visual checkpoints and regression detection. |
| `manage_visual_spec` | `action` ('set' \| 'verify' \| 'list'), `name`?, `screenshot`?, `tolerance`? | Register and verify visual design contract baselines (Visual SDD). |
| `manage_video` | `action` ('ingest' \| 'search' \| 'timeline'), `file_path`?, `query`?, `video_id`? | Ingest WebM/MP4 recordings, search video keyframes, or retrieve timelines. |
| `create_evidence_pack` | `keyframe_state_ids`, `source_video_id`?, `linked_state_memory_nodes`? | Package immutable evidence packs linking video keyframes to state-memory DAGs. |
| `export_trajectories` | `format`? ('json' \| 'llava' \| 'qwen2_vl' \| 'joint'), `trace_id`? | Export multimodal trajectories for model fine-tuning or joint workflow exports. |
| `undo_visual_mutation` | `type`? ('state' \| 'transition' \| 'any') | Revert the last visual state ingestion or transition edge addition. |
| `forget_state` | `state_id` | Purge a specific state and vector embedding for privacy. |
| `wait_for_visual_state` | `target_state_id`, `timeout_ms`? | Poll for target visual state until present or timeout occurs. |

### 3. Agent Permissions & Auto-Run Configuration
To bypass confirmation dialogs when running CLI cache commands or reading/writing brain images, add these allows to your configuration:
* **Google Antigravity (`~/.gemini/config/config.json`)**: Add these rules to your `"globalPermissionGrants"` -> `"allow"` list:
  * `"command(vision-memory-mcp)"` (Allows running any query/ingest command prefix)
  * `"read_file(.*\\.gemini/antigravity/brain/.*)"` (Allows reading brain screenshots)
  * `"write_file(.*\\.gemini/antigravity/brain/.*)"` (Allows saving brain snapshots)

### 4. CLI Commands Reference
Run these commands in the terminal for management and analytics:
* `vision-memory-mcp init [-y|--yes]`: Scaffold workspace .vision-memory-mcp/, .gitignore, .env, and IDE agent rules.
* `vision-memory-mcp init-global`: Re-initialize across all projects registered in ~/.vision-memory-mcp/projects.json.
* `vision-memory-mcp doctor`: Health check storage writability, sharp bindings, Node runtime, and sub-directory Git repos.
* `vision-memory-mcp audit`: Audit sub-directory Git repos, submodules, database locations, and total visual states.
* `vision-memory-mcp inspect`: Display stored visual states in an ASCII table.
* `vision-memory-mcp metrics`: Calculate cache hit rate, token savings, and ROI.
* `vision-memory-mcp view`: Open an interactive force-directed graph visualizer in the browser.
* `vision-memory-mcp export --format [json\|mermaid\|html] --out [file]`: Export the memory graph.
* `vision-memory-mcp prune`: Purge expired or low-access states.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall