n0an/VivaDicta / .agents/skills/axe-simulator-control
axe-simulator-control
VerifiedAutomate iOS Simulator interactions using AXe for taps, gestures, text input, hardware buttons, and video recording
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.4 KB |
| agents/openai.yaml | config | 322 B |
SKILL.md
View on GitHub---
name: axe-simulator-control
description: Automate iOS Simulator interactions using AXe for taps, gestures, text input, hardware buttons, and video recording
---
# AXe Simulator Control
Use this skill when you need to automate iOS Simulator interactions, including taps, gestures, text input, hardware buttons, or video recording.
## Related Skills
- See [`xcodebuild-testing`](../xcodebuild-testing/SKILL.md) for running tests that might require simulator interactions
- See [`screenshot`](../screenshot/SKILL.md) for capturing screenshots from the simulator
## Skill Flow
- Example queries:
- "tap at coordinates (100, 200) in the simulator"
- "type 'hello world' into the simulator"
- "press the home button in the simulator"
- "record a video of the simulator"
- "perform a scroll up gesture"
- Notes:
- AXe is installed via Homebrew: `brew tap cameroncooke/axe && brew install axe`
- Most commands require a simulator UDID (use `axe list-simulators` to find it)
- The simulator must be booted and running for most operations
- AXe uses the iOS Simulator's accessibility framework
- For VivaDicta project, the default simulator is: `iPhone 17 Pro Max, OS=26.4`
- Save screenshots to `llmtemp/screenshots/` directory for temporary test artifacts
### 1. Determine the Required Action
**Touch/Tap Actions** if:
- Need to tap at specific coordinates
- Need to add delays before/after taps
- Need to simulate touch gestures
→ Continue with **Path A: Touch & Tap** (steps 2A-4A)
**Text Input** if:
- Need to type text into the simulator
- Need to test keyboard input
- Need to simulate user text entry
→ Continue with **Path B: Text Input** (step 2B)
**Gestures** if:
- Need to scroll, swipe, or perform system gestures
- Need to test gesture recognizers
- Need to navigate using gestures
→ Continue with **Path C: Gestures** (steps 2C-3C)
**Hardware Buttons** if:
- Need to press home, lock, volume buttons
- Need to test app lifecycle events
- Need to simulate device rotation
→ Continue with **Path D: Hardware Buttons** (steps 2D-3D)
**Video Operations** if:
- Need to record simulator video
- Need to stream simulator output
- Need to capture simulator for documentation
→ Continue with **Path E: Video Operations** (steps 2E-3E)
**List Simulators** if:
- Need to find available simulators
- Need to get a simulator UDID
- First time setup
→ Continue with **Path F: List Simulators** (step 2F)
## Path A: Touch & Tap
### 2A. Get Simulator UDID
```bash
# List available simulators
axe list-simulators
# Store the UDID for the target simulator
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
```
### 3A. Get Accurate Coordinates Using describe-ui
**IMPORTANT:** Always use `axe describe-ui` to get accurate coordinates instead of guessing from screenshots.
```bash
# Get the full UI accessibility tree
axe describe-ui --udid $UDID > ui_tree.json
# Example output for a button:
# {
# "AXFrame" : "{{175, 741}, {52, 50}}",
# "AXLabel" : "stop.circle.fill",
# "type" : "Button",
# ...
# }
# Calculate center coordinates:
# x_center = x + (width / 2) → 175 + (52 / 2) = 201
# y_center = y + (height / 2) → 741 + (50 / 2) = 766
```
**Why this matters:**
- Visual appearance in screenshots doesn't always match accessibility frame coordinates
- UI elements that change state (e.g., record → stop button) may have different positions than they appear
- The accessibility frame is what AXe actually uses for tap detection
### 4A. Perform Tap Action
```bash
# Simple tap at coordinates (use coordinates from describe-ui)
axe tap -x 201 -y 766 --udid $UDID
# Tap with delays before and after
axe tap -x 201 -y 766 --pre-delay 1.0 --post-delay 0.5 --udid $UDID
```
**Available tap options:**
- `-x` and `-y`: Coordinates (required) - get these from `describe-ui` output
- `--pre-delay`: Delay in seconds before tapping
- `--post-delay`: Delay in seconds after tapping
- `--udid`: Simulator UDID (required)
**Note:** The `tap` command does NOT support `--duration` for long press. Use `--pre-delay` and `--post-delay` for timing control.
### 5A. Verify the Action
- Check the simulator visually or via app logs
- If tap didn't register, ensure:
- Simulator is in foreground
- Coordinates are correct (origin is top-left)
- Target element is interactive
- You used coordinates from `describe-ui`, not visual estimates
## Path B: Text Input
### 2B. Type Text into Simulator
```bash
# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# Type text (ensure a text field is focused first)
axe type 'Hello World!' --udid $UDID
# Type text with special characters (use single quotes)
axe type 'user@example.com' --udid $UDID
```
**Notes:**
- The text field must be focused before typing (tap it first if needed)
- Use single quotes to avoid shell interpretation
- Special characters and emojis are supported
## Path C: Gestures
### 2C. Choose Gesture Type
Available gestures:
- `scroll-up` / `scroll-down` / `scroll-left` / `scroll-right`
- `swipe-from-left-edge` / `swipe-from-right-edge`
- `swipe-from-top-edge` / `swipe-from-bottom-edge`
### 3C. Execute Gesture
```bash
# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# Perform scroll gesture
axe gesture scroll-up --udid $UDID
# Perform edge swipe (for navigation)
axe gesture swipe-from-left-edge --udid $UDID
# Perform swipe gesture
axe gesture swipe-from-bottom-edge --udid $UDID
```
## Path D: Hardware Buttons
### 2D. Choose Button Action
Available buttons:
- `home` - Press home button
- `lock` - Press lock/power button
- `volume-up` / `volume-down`
- `siri` - Activate Siri
### 3D. Execute Button Press
```bash
# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# Press home button
axe button home --udid $UDID
# Press lock button with custom duration
axe button lock --duration 2.0 --udid $UDID
# Press volume up
axe button volume-up --udid $UDID
```
**Notes:**
- Lock duration determines press-and-hold behavior
- Home button triggers app backgrounding
- Use for testing app lifecycle events
## Path E: Video Operations
### 2E. Choose Operation Type
**Stream Video** if:
- Need real-time video output
- Building automation tools
- Need MJPEG format
**Record Video** if:
- Need to save video file
- Creating documentation
- Need MP4 format
### 3E. Execute Video Operation
**Stream Video:**
```bash
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# Stream at 10 FPS in MJPEG format
axe stream-video --udid $UDID --fps 10 --format mjpeg > stream.mjpeg
```
**Record Video:**
```bash
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# Record at 15 FPS and save to file
axe record-video --udid $UDID --fps 15 --output recording.mp4
# Stop recording with Ctrl+C
```
## Path F: List Simulators
### 2F. List Available Simulators
```bash
# List all available simulators with their UDIDs
axe list-simulators
# Example output:
# iPhone 17 Pro Max (26.0) - B34FF305-5EA8-412B-943F-1D0371CA17FF
# iPhone 16 (25.0) - A12BC456-7DEF-89GH-IJKL-MNOPQRSTUVWX
```
**Notes:**
- Copy the UDID for use in other commands
- Only booted simulators can be controlled
- Use `xcrun simctl list` for more detailed simulator information
## Common Workflows
### Automated Testing Workflow
```bash
# 1. Get the simulator UDID
UDID=$(axe list-simulators | grep "iPhone 17 Pro Max" | awk '{print $NF}')
# 2. Tap to focus text field
axe tap -x 200 -y 300 --udid $UDID
# 3. Type text
axe type 'test@example.com' --udid $UDID
# 4. Tap submit button
axe tap -x 200 -y 400 --udid $UDID
# 5. Wait and verify (app-specific)
```
### Video Documentation Workflow
```bash
# 1. Get simulator UDID
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"
# 2. Start recording
axe record-video --udid $UDID --fps 30 --output demo.mp4 &
RECORD_PID=$!
# 3. Perform actions
axe tap -x 100 -y 200 --udid $UDID
sleep 1
axe gesture scroll-up --udid $UDID
sleep 1
axe button home --udid $UDID
# 4. Stop recording
kill $RECORD_PID
```
## Known Limitations
### SwiftUI TabView Navigation
**Issue:** SwiftUI's modern `TabView` component (iOS 18+) doesn't expose individual tab buttons in the accessibility hierarchy.
**What you'll see:**
- `axe describe-ui` shows the tab bar as a single `AXGroup` with no children
- Individual tab buttons are NOT listed as separate accessible elements
**Example:**
```json
{
"AXFrame" : "{{0, 791}, {402, 83}}",
"AXLabel" : "Tab Bar",
"type" : "Group",
"children" : [] // ← No individual tab buttons exposed
}
```
**✅ However, tab navigation DOES work with calculated coordinates!**
Even though tab buttons aren't in the accessibility tree, you can still tap them using estimated positions within the Tab Bar frame:
```bash
# Tab Bar frame: x: 0, y: 791, width: 402, height: 83
# Calculate y-center: 791 + (83/2) = 832
# For a 3-tab layout (Record, Notes, Settings):
UDID="D28078F6-0BE9-4EB8-BEBE-BF8EBEA5CA75"
# Tap Record tab (left third)
axe tap -x 103 -y 832 --udid $UDID # x ≈ 402 * 1/4
# Tap Notes tab (center)
axe tap -x 201 -y 832 --udid $UDID # x ≈ 402 * 1/2
# Tap Settings tab (right third)
axe tap -x 268 -y 832 --udid $UDID # x ≈ 402 * 2/3
```
**Key insight:** The Tab Bar frame tells you the tappable area. Calculate positions by:
1. Get y-center from frame: `y + (height / 2)`
2. Divide x-axis by number of tabs to estimate positions
3. Test and adjust x-coordinates based on actual tab spacing
**Alternative workarounds:**
- Use XCUITest for UI testing that requires guaranteed tab detection
- Add custom accessibility identifiers if you control the app code
- Use deep linking or keyboard shortcuts for tab navigation in automation
### Other SwiftUI Components
Some SwiftUI components may have limited accessibility exposure. Always use `axe describe-ui` to verify an element is accessible before attempting to interact with it.
## Troubleshooting
**"Simulator not found" error:**
- Ensure simulator is booted: `xcrun simctl list | grep Booted`
- Verify UDID is correct: `axe list-simulators`
**Tap not registering:**
- **FIRST:** Use `axe describe-ui` to get accurate coordinates (don't guess from screenshots)
- Check coordinates are within screen bounds
- Ensure element is visible and interactive
- Verify the element appears in the accessibility tree with `describe-ui`
- Try adding small delay between taps
**Text not appearing:**
- Tap text field first to focus it
- Check keyboard is visible in simulator
- Verify text encoding with single quotes
**Video recording issues:**
- Ensure sufficient disk space
- Check file permissions in output directory
- Verify FPS is reasonable (10-30 recommended)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |