Skip to content

tough-tongue/toughtongue-skills / skills/ttai-browser-demo-builder

ttai-browser-demo-builder

Verified

Scripts deterministic browser demo steps (tools_config.tools.browser.tool_settings.steps) on an existing Tough Tongue AI Scenario, so its voice agent clicks exact XPath selectors instead of improvising. Signs the demo browser in with ttai:authenticate_browser, converts a pasted browser recording or a walked-through flow into steps, saves them with ttai:update_scenario without losing the browser profile, and repairs steps that click the wrong element. Use when the user says "record browser demo steps", "turn my recording into steps", "make my demo deterministic", "the demo clicks the wrong thing", or "log the demo browser in". Not for creating the Scenario (ttai-agent) or analyzing sessions (ttai-session-analyst).

references/steps-format.md

View on GitHub
# Steps Format — Schema, Methods, Payload, Instructions

## Contents

- Schema
- Methods
- Read-merge-write (the only safe write)
- Placeholders
- BROWSER TOOL INSTRUCTIONS block
- Flow phase template
- Fallback steps

---

## Schema

Location: `tools_config.tools.browser.tool_settings`.

```json
{
  "initialUrl": "https://app.example.com/",
  "contextId": "<kept from the read — never invented>",
  "steps": {
    "<step_name>": {
      "description": "One sentence: the screen transition this step performs",
      "actions": [
        {
          "description": "Standalone name of the target element",
          "method": "click",
          "selector": "xpath=//button[@aria-label='Open reports']",
          "arguments": []
        }
      ]
    }
  }
}
```

- Step name (the key): what the agent passes as `name` in `command: "step"`.
  Short, stable, intent-named.
- Action `description`: the self-heal prompt when the selector fails. Must
  identify the element with no other context. Static — no placeholders.
- `arguments`: always an array of strings, `[]` when unused.
- The server does not validate steps. A typo in `method` or a malformed selector
  is only discovered at replay.
- The Scenario editor shows `click`, `fill`, `scrollTo` in its method menu but
  keeps any other valid method you write.

## Methods

Each method with its `arguments`:

- `click` `[]` — buttons, tabs, cards, links, menu items.
- `fill` `["text"]` — clears, then types into an input; fires input events.
- `type` `["text"]` — types without clearing.
- `press` `["Enter"]` — key press on the focused element; put it right after a
  fill.
- `selectOption` `["Option label"]` — native `<select>` only; custom dropdowns =
  two clicks.
- `hover` `[]` — reveal hover menus.
- `doubleClick` `[]` — double-click targets.
- `dragAndDrop` `["xpath=<target>"]` — drag selector onto the target XPath.
- `scrollIntoView` `[]` — bring an element on screen as a step's visible
  "scroll".
- `scrollTo` `["50%"]` — scroll an element's OWN scrollbar to a percentage.
- `nextChunk` / `prevChunk` `[]` — scroll an element by its own height (`/html`:
  one viewport).

`scrollTo` is not scroll-into-view. On a normal element it scrolls that
element's internal scrollbar (often a no-op). On `xpath=/html` it scrolls the
page to the percentage. Replay already scrolls XPath targets into view before
each action, so a `scrollTo` before a click is dead weight — delete it.

## Read-merge-write (the only safe write)

`ttai:update_scenario` merges per tool, shallowly: fields you send on
`tools.browser` replace the stored ones, and `tool_settings` is replaced as a
whole object. Sending `{"steps": {...}}` alone deletes `contextId` (the
platform's opaque id for the Scenario's browser profile) and `initialUrl`.
Copy `contextId` back unchanged.

1. Read: `ttai:get_resource(scenarios, id)` (pass `org_id` for org
   Scenarios). Take `tools_config.tools.browser.tool_settings` verbatim.
2. Merge in memory: set or replace only your step keys under `steps`. Keep every
   other key and step unchanged.
3. Write the complete object:

```json
{
  "scenario": {
    "id": "<SCENARIO_ID>",
    "tools_config": {
      "tools": {
        "browser": {
          "tool_settings": {
            "initialUrl": "<from the read>",
            "contextId": "<from the read, unchanged>",
            "steps": {
              "<existing_step>": { "...": "unchanged from the read" },
              "<new_step>": { "description": "...", "actions": [] }
            }
          }
        }
      }
    }
  }
}
```

4. Re-read and diff: `contextId` identical, all step keys present.

`should_register` and `add_to_system_prompt` survive when omitted; only the keys
you send on `tools.browser` are replaced.

## Placeholders

- `%name%` in `selector` or `arguments` is replaced from the step call's
  `variables` (a JSON object string, e.g. `{"customer": "Acme"}`).
- A step whose placeholders are not all supplied is rejected before running.
- Each placeholder is a value the voice agent must produce mid-demo. Prefer
  fixed demo data. List every placeholder and its source in the instructions.

## BROWSER TOOL INSTRUCTIONS block

Add to `ai_instructions`:

```text
## BROWSER TOOL INSTRUCTIONS
The browser is pre-opened at [INITIAL_URL]. Use `runBrowserCommand`:
- `command: "step"` + `name` — runs a pre-recorded batch. PRIMARY command.
- `command: "goto"` + `url` — direct navigation (use between steps).
- `command: "capture"` — screenshot (EXPENSIVE — marked milestones only).
- `command: "observe"` / `command: "act"` — off-script moments only.

Speed rules:
- Always use `step` for the scripted flow. Never rebuild it with act.
- Steps run in the background — keep narrating while they execute.
- One browser command at a time: wait for the completion message before the
  next, or it is rejected.
- If a step FAILS you are told which action broke: `capture`, then recover
  with `act`.
```

## Flow phase template

```text
### Phase N: <Name>
1. Say: "<one narration line>"
2. `goto` "<url>"            (only if this phase starts on a new page)
3. `step` name: "<step_name>"
   - If it fails at "<first action description>": run step "<fallback>".
4. Narrate while it runs: "<what appears on screen>"
   — or — `capture` — MILESTONE N: <what to confirm>.
```

## Fallback steps

When one screen has two states (a cookie banner, a dismissible tour, an optional
panel), record a step per state and route on the failure message:

```text
3. `step` name: "open_reports"
   - If it fails at "Dismiss button on the product tour popup", run step
     "open_reports_no_tour" instead.
```

A login form appearing means the browser profile is signed out. Stop and
have the user sign in again through `ttai:authenticate_browser`; never record
steps that type a username or password.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall