Skip to content

sumitaich1998/jarvisvr / skills/stage/annotate-reality

annotate-reality

Verified

Place world-anchored labels, callouts, pins, and boxes onto real objects and locations so information sticks to the physical world. Use for labeling things in the room, pinning a spot, boxing a detected object, or step-by-step "point at the X" guidance. Triggers: label this, pin here, mark that, point to, highlight, call out, annotate, put a label on.

Annotate Reality

Anchor information to the physical world. The perception-agent decides what a real thing is; you render the spatial label/box/pin in the right place.

Pick the annotation

GoalWidgetTool
Name a real object with a calloutvision_annotationannotate_object
Box a detected object's extentbounding_box_3ddraw_bounding_box
Drop a pin/marker at a spotscene_labeldrop_scene_label
Free-floating heading/captiontext_labelshow_text

Steps

  1. Get the target position (world meters) from the perception annotation, gaze hit point, or a scene surface.
  2. Choose the widget from the table; keep labels short.
  3. Anchor in world with billboard:true so the label faces the user; lift the callout slightly above the object (e.g. +0.15 m in Y) with a leader_line.
  4. Spawn, then keep it anchored as the user moves (world anchor handles this).
  5. Clean up stale annotations via close_hologram when no longer relevant.

Output

vision_annotation (annotate_object, props per registry.json):

{ "widget_type": "vision_annotation",
  "transform": { "anchor": "world", "position": [0.3,0.95,0.7], "billboard": true },
  "props": { "label": "coffee mug", "confidence": 0.78, "detail": "ceramic, ~350 ml",
             "leader_line": true, "target_position": [0.3,0.8,0.7], "color": "#7FE7FF" },
  "interactions": ["tap","grab","dwell"] }

bounding_box_3d (draw_bounding_box):

{ "widget_type": "bounding_box_3d",
  "props": { "label": "laptop", "confidence": 0.91, "size": [0.33,0.02,0.23], "color": "#FFB74D" } }

scene_label (drop_scene_label):

{ "widget_type": "scene_label",
  "props": { "text": "Keys (last seen here)", "icon": "pin", "color": "#FF5252", "pin": true } }

Edge cases

  • No reliable position → fall back to a head-anchored text_label and say you couldn't pin it precisely.
  • Many annotations → cap to the relevant few; over-labeling clutters reality (hand to declutter-space).
  • Moving target → annotations are static once placed; re-place if the object moves.
  • Confidence honesty → carry the real confidence so the UI can style uncertainty.
  • Steps/tour → sequence pins and advance as the user completes each.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall