Skip to content

sumitaich1998/jarvisvr / skills/stage/annotate-reality

annotate-reality

Verified

Place world-anchored labels, callouts, pins, and boxes onto real objects and locations so information sticks to the physical world. Use for labeling things in the room, pinning a spot, boxing a detected object, or step-by-step "point at the X" guidance. Triggers: label this, pin here, mark that, point to, highlight, call out, annotate, put a label on.

FileKindSize
SKILL.mdskill md3.0 KB
---
name: annotate-reality
description: >-
  Place world-anchored labels, callouts, pins, and boxes onto real objects and
  locations so information sticks to the physical world. Use for labeling things
  in the room, pinning a spot, boxing a detected object, or step-by-step "point at
  the X" guidance. Triggers: label this, pin here, mark that, point to, highlight,
  call out, annotate, put a label on.
license: MIT
compatibility: >-
  Requires JarvisVR agent-backend stage runtime; world anchoring benefits from
  perception/scene data. Perception widgets per PROTOCOL.md §8.5.
metadata:
  agent: stage-agent
  category: stage
  version: "1.0"
  author: jarvisvr
allowed-tools: annotate_object drop_scene_label draw_bounding_box show_text
---
# Annotate Reality

Anchor information to the physical world. The perception-agent decides *what* a
real thing is; you render the spatial label/box/pin in the right place.

## Pick the annotation

| Goal | Widget | Tool |
| ---- | ------ | ---- |
| Name a real object with a callout | `vision_annotation` | `annotate_object` |
| Box a detected object's extent | `bounding_box_3d` | `draw_bounding_box` |
| Drop a pin/marker at a spot | `scene_label` | `drop_scene_label` |
| Free-floating heading/caption | `text_label` | `show_text` |

## Steps

1. **Get the target position** (world meters) from the perception annotation,
   gaze hit point, or a scene surface.
2. **Choose the widget** from the table; keep labels short.
3. **Anchor in `world`** with `billboard:true` so the label faces the user; lift
   the callout slightly above the object (e.g. +0.15 m in Y) with a `leader_line`.
4. **Spawn**, then keep it anchored as the user moves (world anchor handles this).
5. **Clean up** stale annotations via `close_hologram` when no longer relevant.

## Output

`vision_annotation` (`annotate_object`, props per registry.json):

```json
{ "widget_type": "vision_annotation",
  "transform": { "anchor": "world", "position": [0.3,0.95,0.7], "billboard": true },
  "props": { "label": "coffee mug", "confidence": 0.78, "detail": "ceramic, ~350 ml",
             "leader_line": true, "target_position": [0.3,0.8,0.7], "color": "#7FE7FF" },
  "interactions": ["tap","grab","dwell"] }
```

`bounding_box_3d` (`draw_bounding_box`):

```json
{ "widget_type": "bounding_box_3d",
  "props": { "label": "laptop", "confidence": 0.91, "size": [0.33,0.02,0.23], "color": "#FFB74D" } }
```

`scene_label` (`drop_scene_label`):

```json
{ "widget_type": "scene_label",
  "props": { "text": "Keys (last seen here)", "icon": "pin", "color": "#FF5252", "pin": true } }
```

## Edge cases

- **No reliable position** → fall back to a `head`-anchored `text_label` and say
  you couldn't pin it precisely.
- **Many annotations** → cap to the relevant few; over-labeling clutters reality
  (hand to `declutter-space`).
- **Moving target** → annotations are static once placed; re-place if the object
  moves.
- **Confidence honesty** → carry the real `confidence` so the UI can style
  uncertainty.
- **Steps/tour** → sequence pins and advance as the user completes each.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall