Skip to content

AlexAnsart/demo-studio / skills/sync-narration

sync-narration

Verified

Syncs a separately-recorded narration track onto a silent screen recording — pads audio with calculated pauses to match on-screen pacing (no voice during cursor movement), then burns styled captions. Use when the user provides a silent screen-recorded video plus a narration .mp3 and asks to sync/combine/caption them.

---
name: sync-narration
description: Syncs a separately-recorded narration track onto a silent screen recording — pads audio with calculated pauses to match on-screen pacing (no voice during cursor movement), then burns styled captions. Use when the user provides a silent screen-recorded video plus a narration .mp3 and asks to sync/combine/caption them.
---

# Sync narration onto a silent screen recording

Input: silent screen recording (from `film-demo`) + a narration `.mp3` (from
`narrate`, or any TTS/recorded voice). Output: final `.mp4` with paced
narration + burned-in captions.

**Never ship** with (a) captions out of sync with audio, (b) long video
stretches with no narration at the start or end — that reads as broken/empty,
(c) clipped TTS onsets **or** release tails (last syllable cut off — use
`--seg-postroll`, see below), or (d) narration playing while the cursor is
still moving — voice should land only after the pointer has settled on the
target element.

---

## Quick start (copy-paste per video)

```bash
# 0 — once
pip install -r scripts/requirements.txt

# 1 — align audio to script (local Whisper, no API key)
python scripts/align-narration.py \
  demos/<slug>/narration.mp3 \
  --script-file script.txt \
  --out /tmp/aligned.json

# 2 — contact sheet → pick segmentTargets (when each sentence should start on screen)
ffmpeg -y -i demos/<slug>/no-sound.mp4 -vf "fps=1/2,scale=384:-1,tile=5x6" -update 1 -frames:v 1 /tmp/contact.png

# 3 — write cues.txt (one caption line per readable cue, exact script wording, ≤ 60 chars)
# 4 — build plan (defaults: seg-preroll 0.12, seg-postroll 0.20, min-silence 0.22, max-cue-chars 80)
node scripts/build-plan-from-align.js \
  --align /tmp/aligned.json --out /tmp/plan.json \
  --video /tmp/work.mp4 \
  --audio demos/<slug>/narration.mp3 \
  --output demos/<slug>/final.mp4 \
  --targets 0,2,8,… \
  --cues-file cues.txt \
  --tail-pad 1.0

# 5 — work video (default: full no-sound.mp4, no trim)
ffmpeg -y -i demos/<slug>/no-sound.mp4 -c:v copy -an /tmp/work.mp4
# Only trim if you verified dead footage — see "Video trim (conservative)" below

# 6 — validate + render
node scripts/sync-narration.mjs /tmp/plan.json --validate-only
node scripts/sync-narration.mjs /tmp/plan.json

# 7 — spot-check narration ↔ picture on the FINAL render (mandatory)
node scripts/spot-check.mjs /tmp/plan.json
# → read EVERY PNG in spot-check/: at each sentence start the screen must already
#   show what the sentence talks about; during long gaps something must be happening.
```

---

## How the three layers fit together

| Layer | Tool | What it controls |
|-------|------|------------------|
| **When each word is spoken** | `align-narration.py` (faster-whisper) | Word + sentence timestamps in the raw `.mp3` |
| **When each sentence lands on screen** | `segmentTargets` in `build-plan-from-align.js` | Inserted gaps *between* sentences |
| **When each caption appears** | Caption `start`/`end` from word alignment | Burned subs locked to audio |
| **Audio attack (no clipped onset)** | `--seg-preroll` (default **0.12 s**, silence-aware) | `atrim` starts slightly *before* Whisper word onset |
| **Audio release (no clipped ending)** | `--seg-postroll` (default **0.20 s**, silence-aware) | `atrim` extends after last word; final segment runs to mp3 end |
| **No cut inside a word** | orphan-word absorption + `--min-silence` (default **0.22 s**) | boundary words are absorbed into the earlier sentence; sentences without usable silence between them are **merged into one segment** (one target for the block) |

**Gaps** stretch narration to match UI pacing. **Video length** defaults to the full silent
recording — the padded audio track is often shorter than the screen capture, and the tail of the
video carries visual payoffs after the last spoken word.

---

## 1. faster-whisper alignment (no API key)

Install: `pip install -r scripts/requirements.txt`

Default model is `base` (~150 MB, one-time download) — its word boundaries are noticeably
tighter than `tiny`; clipped onsets (first letter of a word missing) usually trace back to
running `tiny`. Only drop to `--model tiny` when iterating fast on something non-final.

```bash
python scripts/align-narration.py narration.mp3 \
  --script-file script.txt \
  --out aligned.json
```

Output:
- `words[]` — `{ word, start, end }` for every token
- `sentences[]` — sentence blocks matched to the script

**Use this for:**
- Audio `segments` boundaries (one per sentence — never split at comma pauses)
- Caption cue `start`/`end` (via `build-plan-from-align.js` + `cues.txt`)

**Do not** use `silencedetect` alone for caption timing — comma pauses inside a sentence are
not sentence boundaries.

### Word-safe segment boundaries (fixes clipped attack/release AND mid-word cuts)

Whisper word timestamps lag the audible envelope on both sides (~50–350 ms), and its sentence
bounds sometimes leave a split word's tail token stranded between two sentences — an inserted gap
there cuts the word in half in the output. `build-plan-from-align.js` handles all of this
deterministically:

1. **Orphan-word absorption** — any word that ends before the next sentence starts is absorbed
   into the earlier sentence. A sentence's audio runs until the next one begins; nothing is left
   in no-man's land.
2. **Tight-boundary merge** (`--min-silence`, default **0.22 s**) — when the silence between two
   sentences is smaller than this, cutting there would clip the TTS release/attack, so the two
   sentences are **merged into one audio segment**. The script prints which sentences merged;
   pass **one** `--targets` value for the merged block.
3. **Silence-aware padding** — `--seg-preroll` (default **0.12**) and `--seg-postroll` (default
   **0.20**) extend each cut into the surrounding silence, but never more than **half** of it
   (the neighbouring sentence keeps its share) and the final segment always runs to the full
   `.mp3` duration.

The padded bounds are **baked into `plan.segments`** (`segmentPreRoll`/`segmentPostRoll` are 0
in the emitted plan) — `sync-narration.mjs` uses them as-is.

- **Listen-check before ship:** scrub the last 1 s of audio in the rendered file — the final word
  should decay naturally, not stop mid-syllable.

---

## 2. Contact sheet → segmentTargets

```bash
ffmpeg -y -i no-sound.mp4 -vf "fps=1/2,scale=384:-1,tile=5x6" -update 1 -frames:v 1 contact.png
```

Read the grid (cell `i` ≈ `i×2` s). For each sentence in `aligned.json`, pick the output time
when that topic is **visible** — pass as comma list to `--targets`.

### No voice during cursor movement

**Do not start a sentence while the pointer is traveling.** Narration competes with motion and
reads as rushed or out of sync. For each sentence:

1. Find the frame where the cursor **lands** on the element you are describing — not the frame
   where movement begins.
2. Set `--targets[i]` to that settled moment (add gap before the sentence if the raw `.mp3`
   would speak too early).
3. If a beat requires both travel *and* a click, speak **after** the click completes and the UI
   has updated — never over the travel or click animation.

When in doubt, pause 0.5–1 s longer before the line rather than talking over cursor motion.

---

## 3. Captions (`cues.txt`)

One line per readable cue — **≤ 60 characters preferred, 80 hard max** (the plan builder fails
above `--max-cue-chars`, default 80; a longer line physically overflows the caption bar at the
default fontsize). Split long sentences at natural clause breaks; lines starting with `only` /
`same` get an em dash prefix in the plan builder.

Compound sentence = **one audio segment**, multiple caption lines. Whisper word times set each
cue's `[start,end]` inside that segment.

---

## 4. Build plan + trim video

```bash
node scripts/build-plan-from-align.js \
  --align aligned.json --out plan.json \
  --video work.mp4 --audio narration.mp3 --output final.mp4 \
  --targets 0,2,8,16,19.64,24.42,28 \
  --cues-file cues.txt \
  --tail-pad 1.0
```

Prints: `speech ends at Xs → trim video to Ys (tail-pad 1s)` — plus any sentence merges
(then `--targets` needs one value per **merged** segment, not per script sentence).

**Important:** that `expectedVideoDuration` is the **padded audio length**, not a mandate to cut
the screen recording. If the silent video is longer, keep it — the extra tail is usually the
visual payoff after the last word (see validate warning: `video is longer than built timeline` → OK).

Plan fields:
- `segments` — `[start,end]` in **original mp3** time, word-safe padding already baked in
- `segmentPreRoll` / `segmentPostRoll` — **0** in emitted plans (padding baked into `segments`)
- `audioDuration` — full `.mp3` length from alignment (used for final-segment tail)
- `gaps` — silence inserted **between** segments so each starts at `--targets[i]`
- `trailGap` — `--tail-pad` (default **1.0 s** after last word)
- `expectedVideoDuration` — padded audio length (`speech end + tail-pad`); informational only
- `segmentTargets` / `segmentTexts` — echoed for `spot-check.mjs`

### Video trim (conservative — default: **no trim**)

**Start from the full silent video.** The sync script accepts a video longer than the padded
audio; output duration follows the video. After the last caption, the viewer should still see
key beats from the beat script.

Only cut when you have **verified** the removed section adds nothing:

1. **Before any cut** — re-read the beat script and every planned narration clause.
2. **Contact sheet** — generate one for the full recording; map each beat to a cell/time.
3. **Decide** — if a proposed cut removes any mapped beat, **do not cut**.
4. **Spot-check** — grab frames at the proposed cut point and 2 s after the planned end:
   `ffmpeg -y -ss <t> -i no-sound.mp4 -update 1 -frames:v 1 check.png`

| Situation | Action |
|-----------|--------|
| **Default** | `-c:v copy` the full silent video → `work.mp4`; ignore `expectedVideoDuration` |
| Video longer than padded audio (validate warning) | **Expected** — keep the video; tail holds visual payoff |
| Dead tail *after* the last beat (cursor idle, blank screen, stop-recording artefact) | Trim only that dead tail (typically ≤ 1–2 s) |
| Dead head *before* the first beat (accidental pre-roll) | `-ss` head trim only if the first 2+ s contain **no** beat content |
| `trailGap` > 1.5 s in validate output | Wrong gap/targets — fix `--targets`, do not inflate `trailGap` to fill dead video |
| Tempted to trim to `expectedVideoDuration` to "tighten" runtime | **Stop** — check whether that removes a payoff first |

**Never** combine head trim + tail trim to match audio length without cross-checking every beat.
When in doubt, ship the full recording. Keep the silent master and `.mp3` untouched — only the
final `.mp4` ships.

---

## 5. Validate + render

```bash
node scripts/sync-narration.mjs plan.json --validate-only   # mandatory
node scripts/sync-narration.mjs plan.json
```

Read the printed timeline. Every caption window must sit inside a speech block. `trail silence`
should be ≤ `--tail-pad` (≈ 1 s).

A validate warning like `video is longer than built timeline` is **normal** when keeping the full
recording for post-narration visual payoffs — confirm the extra tail matches a beat, then render.

Verify sync **and** payoff — mandatory, on the FINAL render:

```bash
node scripts/spot-check.mjs plan.json
```

It extracts a frame at the start and end of **every** audio segment plus the midpoint of every
long gap, and writes `spot-check/spot-check.json`. Read **every** PNG:

- At a sentence's start frame, the screen must **already show** what the sentence talks about —
  never narrate a result ("it's saved") while a spinner is still visible.
- The pointer should be **idle** on the described element, not mid-travel.
- During a long gap frame, something must visibly be happening (otherwise tighten the gap).

Re-run `build-plan-from-align.js` with later `--targets` if any line starts too early.

---

## Caption style & fonts

Default look (override any field via `plan.style`): full-width black bar at bottom
(`barColor: 'black@0.82'`), 1px white top border, centered monospace, no drop shadow.

`sync-narration.mjs` auto-detects a system monospace font (Windows: Consolas; macOS: Menlo;
Linux: DejaVu Sans Mono / Liberation Mono). If none is found on your system, set
`plan.style.fontfile` to an absolute path to a `.ttf`/`.ttc` file.

---

## Manual plan.json (fallback)

Only if Whisper cannot run. Same rules: sentence segments only, gaps between sentences,
captions via `{ text, segment, start, end }`, `trailGap` ≤ 1.5 s, `segmentPreRoll` 0.12 s,
`segmentPostRoll` 0.20 s (unless ending still clips), never cut where silence < 0.22 s,
trim video to match.

---

## ffmpeg gotchas (cross-platform)

- Pass `-filter_complex` via `spawnSync` args — not `-filter_complex_script` with raw drive
  paths on Windows.
- Windows only: escape drive colons in filter values (`C\\:/Windows/...`) — `sync-narration.mjs`
  does this automatically via `ffPath()`.
- Use `textfile=` for captions, never `text=` (apostrophes/colons break the graph).
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall