puntorigen/avatar-skills / audio-theater
audio-theater
VerifiedTurn a dialogue, a story with dialogue, or a one-line idea into multi-character audio using Gemini TTS, with word-level timecoded transcription (WhisperX or local faster-whisper), realistic ambient/foley SFX (ElevenLabs or the sound-effects skill), and instrumental score/music (MiniMax Music 2.6). Optional stereo spatialization (pan + distance + movement) and an ffmpeg mix with sidechain ducking (music ducks under voices+SFX). Three modes - theater (dramatized radio play), lipsync (clean per-line clips <=15s plus a manifest for the seedance-2 skill), and podcast (two hosts with intro/bed/outro music). With music it delivers three tracks (full mix, music-only, and a no-music stem for animated storyboards) plus a timecoded transcript and per-line voice + SFX/music files. Use for an audio drama, audio theater / audioteatro / radioteatro, a dramatized dialogue, a score for a story/podcast/storyboard, voiceover for lipsync, a TTS conversation, or a podcast episode from a script or idea.
| File | Kind | Size |
|---|---|---|
| REFERENCE.md | markdown | 26.9 KB |
| SKILL.md | skill md | 26.9 KB |
| scripts/_common.py | script | 16.8 KB |
| scripts/audio_music.py | script | 10.3 KB |
| scripts/build_transcript.py | script | 4.8 KB |
| scripts/elevenlabs_sfx.py | script | 3.8 KB |
| scripts/export_lipsync.py | script | 4.4 KB |
| scripts/generate_sfx.py | script | 10.0 KB |
| scripts/generate_voices.py | script | 16.6 KB |
| scripts/mix.py | script | 10.0 KB |
| scripts/mix_spatial.py | script | 23.0 KB |
| scripts/requirements.txt | manifest | 364 B |
| scripts/setup_key.py | script | 4.2 KB |
| scripts/spatial.py | script | 6.8 KB |
| scripts/split_tracks.py | script | 8.3 KB |
| scripts/transcribe.py | script | 8.8 KB |
| scripts/voices.json | config | 2.1 KB |
| scripts/write_script.py | script | 9.9 KB |
REFERENCE.md
View on GitHub# Audio Theater - Reference
Detailed reference for the `audio-theater` skill. The main workflow lives in [SKILL.md](SKILL.md).
## Scripts at a glance
| Script | Reads | Writes |
|--------|-------|--------|
| `write_script.py` | idea or dialogue `.txt` | `script.json`, `story.md` |
| `generate_voices.py` | `script.json` | `lines/*.wav`, `dialogue.wav`, `lines.json` |
| `export_lipsync.py` | `lines.json` | `lipsync.json` |
| `transcribe.py` | `dialogue.wav` (+ `lines.json`) | `words.json` |
| `generate_sfx.py` | `cues.json` (you author it) | `sfx/*.mp3` (updates `cues.json`) |
| `mix.py` | `dialogue.wav` + `cues.json` | `final.mp3` (+ `final.nomusic.mp3` / `final.music.mp3` stems when there's music) |
| `mix_spatial.py` | `lines.json` + `script.json` (seats/spatial) + `cues.json` (spatial) | `final.mp3` (stereo positioned, theater only) + same stems |
| `spatial.py` | — (helper imported by `mix_spatial.py`) | pan/distance/movement ffmpeg chains |
| `split_tracks.py` | `lines.json` + `cues.json` (+ `script.json` roles) | `narration.mp3`, `lipsync_mix.mp3`, `tracks.json` |
| `build_transcript.py` | `lines.json` / `words.json` / `cues.json` | `transcript.md` |
All scripts take `--out <project-folder>` and default secondary paths to files inside it.
## Keys
Resolution order (first hit wins), implemented in `scripts/_common.py`:
- **Gemini**: `GEMINI_API_KEY` / `GOOGLE_API_KEY` env -> `audio-theater/config.json` -> `asset-generator/config.json` -> `avatar-video-reel/config.json`.
- **Replicate**: `REPLICATE_API_TOKEN` env -> `audio-theater/config.json` -> `sound-effects` -> `avatar-video-reel` -> `bg-music-hq` -> `bg-music` -> `brand-asset-studio` config.
- **ElevenLabs** (optional, realistic SFX backend): `ELEVENLABS_API_KEY` / `ELEVEN_API_KEY` env -> `audio-theater/config.json` (`elevenlabs_api_key`) -> `bg-music` / `bg-music-hq` config.
`setup_key.py` (no args) imports keys into `config.json` without overwriting; `--show` prints masked values + the resolved SFX backend; `--gemini` / `--replicate` / `--elevenlabs` set them explicitly.
## SFX backend (`generate_sfx.py --backend`)
Ambient/oneshot cues are generated by one of two backends (`--backend auto|elevenlabs|sound-effects`; `auto` = elevenlabs if a key exists, else sound-effects; default also from `config.json:default_sfx_backend`):
- **`elevenlabs`** — `POST https://api.elevenlabs.io/v1/sound-generation`, model `eleven_text_to_sound_v2` (raw HTTP, no SDK dep; see `elevenlabs_sfx.py`). Body: `{text, model_id, loop, prompt_influence, duration_seconds?}`, `output_format` as query param (default `mp3_44100_128`). `duration_seconds` must be 0.5-30 (or omitted = model auto-length). Ambient cues are requested with `loop=true` at up to `EL_AMBIENT_LOOP_SECONDS` (22s) and `mix.py` repeats the seamless loop across the window. Best realism for foley/nature.
- **`sound-effects`** — shells to the `sound-effects` skill (Stable Audio 2.5); uses the cue `category`. Synthesis quality; good for abstract/UI, weaker on realistic foley.
`music` cues use the music backend (`--music-backend auto|hq|fast`; default also from `config.json:default_music_backend`), independent of `--backend`. See "Music & stems" below.
## Gemini TTS
- Default model `gemini-3.1-flash-tts-preview` (also valid: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts`). Override with `generate_voices.py --model` or `config.json:default_tts_model`.
- API: `client.models.generate_content(model, contents, config)` with `response_modalities=["AUDIO"]` and a `SpeechConfig`. Single-speaker uses `voice_config`; two-speaker uses `multi_speaker_voice_config` (**max 2 speakers**).
- Output is **PCM 24 kHz, 16-bit, mono** in `response.candidates[0].content.parts[0].inline_data.data`; `_common.pcm_to_wav` wraps it into WAV.
- Known issues handled by the skill: the model occasionally returns text tokens (random ~500) -> `generate_voices.py` retries up to 4x; vague prompts can make it read instructions aloud -> we use a "Synthesize speech... read ONLY the line after LINE:" preamble.
- Quality drifts past a few minutes -> generate per line (the default), not one giant call.
### The 30 voices
Listed with their character in `scripts/voices.json`. Short list: Zephyr, Puck, Charon, Kore, Fenrir, Leda, Orus, Aoede, Callirrhoe, Autonoe, Enceladus, Iapetus, Umbriel, Algieba, Despina, Erinome, Algenib, Rasalgethi, Laomedeia, Achernar, Alnilam, Schedar, Gacrux, Pulcherrima, Achird, Zubenelgenubi, Vindemiatrix, Sadachbia, Sadaltager, Sulafat.
### Audio tags
Inline `[tag]` modifiers steer delivery, e.g. `[whispers]`, `[shouting]`, `[laughs]`, `[sighs]`, `[excited]`, `[tired]`, `[sarcastic]`, `[crying]`, `[gasp]`. Put them in a line's `tags` array (woven in automatically) or directly inside `text`. Use English tags even for non-English transcripts. The language is auto-detected from the transcript.
**Flaky tags / no-tags fallback.** Some tags (notably `[whispers]`) make the TTS model return text tokens instead of audio, deterministically failing every retry. `generate_voices.py` handles this automatically: if a line (or multi-speaker chunk) fails all retries *and* it has tags, it retries once with the tags stripped, so a style note never drops a line — you just lose that one effect. If you specifically need a whisper, prefer a softer-but-reliable tag like `[softly]`/`[quietly]`, or accept the plain fallback.
## Pacing & fixed-duration targets
The Gemini voices read **slowly and deliberately** and pad each clip with leading/trailing/inter-sentence silence — a 5-line script can come out ~24-25s raw. When the audio has to hit a fixed length (a 15s reel, an ad beat), fit it with **script length, not speed**:
- **Budget ~2 spoken words/sec.** That already accounts for the deliberate pace + the `pause_after` gaps, so `target_seconds × 2` ≈ the total word budget (15s ≈ 25-30 words). Split it into the **fewest, shortest lines** that carry the message.
- **Fewer lines > faster speech.** For ads/explainers the goal is *low cognitive load*; rushed VO hurts comprehension. If you're over budget, **drop or merge a line** before touching speed.
- **Trim dead air first.** Most of the overage is silence, not speech, and trimming it recovers real seconds without changing perceived pace (trim head/tail + collapse long internal gaps):
`silenceremove=start_periods=1:start_threshold=-40dB:start_silence=0.03:stop_periods=-1:stop_threshold=-40dB:stop_silence=0.14:detection=peak`
- **Cap `atempo` at ~1.1×.** Only after trimming, if still slightly long, apply a gentle uniform `atempo` (pitch-preserving). Past ~1.1-1.15× it sounds hurried — never use heavy stretching to rescue an over-long script.
- If you do time-scale per-line clips, **rebuild `dialogue.wav` and `lines.json`** from the processed clips (re-pad each with `pause_after`, re-concat, and rescale `start`/`end`/`duration`) so `cues.json`, the transcript and `lipsync.json` stay in sync. Better: avoid the need by budgeting the script up front.
- `[whispers]`, ellipses (`…`) and heavy emotional tags **slow delivery further** and add pauses — drop them when fighting for time (and `[whispers]` is flaky anyway; see Audio tags).
## WhisperX (Replicate) output
`transcribe.py` calls `victor-upmeet/whisperx` with `align_output=true`, `diarization=false`, and `language` (ISO code). Output is `{"segments": [...], "detected_language": "..."}`; with alignment each segment has `words: [{word, start, end}]`. The local fallback `faster-whisper` (`word_timestamps=True`) produces the same normalized shape. Result `words.json`:
```json
{
"backend": "replicate",
"language": "es",
"duration": 12.4,
"word_count": 30,
"words": [{"word": "La", "start": 0.02, "end": 0.18, "line_index": 0}],
"segments": [{"start": 0.0, "end": 3.2, "text": "...", "words": [ ... ]}]
}
```
`line_index` is attached when `lines.json` has per-line timings (per-line TTS mode).
## cues.json authoring
You (the agent) write `cues.json` after reading `script.json` + `words.json`. Anchor one-shots to word/line timecodes. Fields:
- `id` (string, unique; becomes the sfx filename and the `[SFX: id]` placeholder).
- `type`: `ambient` (looping bed, ducked under voice), `oneshot` (single hit at `start`), `music` (instrumental bed/jingle via MiniMax Music 2.6, ducked under content).
- `description`: natural-language prompt for the generator. For realism, structure it as **event → material/source → motion → environment** and name concrete materials (`dry leaves`, `rusty hinges`, `popping embers`); one source per cue. **For repeated/continuous actions describe the full sequence** ("several footsteps crunching", "a few twigs snapping in succession") with a matching `gen_seconds` (~3-5s), or the model renders a single thin hit. For a dramatized feel, stylize the prompt ("mystical eerie wind with an otherworldly hum") and lower `prompt_influence`. Avoid vague filler (`epic`, `high quality`) and musical words. (The sound-effects backend also auto-appends recording-context keywords from the category preset.)
- `category` (ambient/oneshot, **sound-effects backend only**; ignored by elevenlabs): sound-effects category — pick by the *sound's nature*, not the cue type: `foley` for physical/man-made hits (footsteps, doors, twig snaps), `nature` for organic/outdoor sources (wind, fire, animals, water), `ambient` for abstract atmospheric beds, plus `mechanical`, `transition`, `game`, `voice`, `generic`.
- `prompt_influence` (ambient/oneshot, **elevenlabs backend only**): 0-1, default 0.4. Lower = more natural variation; higher = follows the prompt more strictly.
- `mood` (music): a mood from the `bg-music-hq` library (read-only) used to steer the prompt — `podcast-intro`, `podcast-bed`, `podcast-outro`, `cinematic`, `ambient`, `uplifting`, `pet-lullaby`, etc. (`audio_music.py --list-moods`).
- `start`, `end` (seconds). `oneshot` may omit `end`.
- `gain_db` (level offset). Dialogue stays on top: SFX present but **not louder than the voices** — one-shots `-12`..`-8` (peaky door/gust/snap ~`-13`..`-11`), beds `-18`..`-15`; **music** base `-24`..`-20`. Mixer defaults when omitted: oneshot `-9`, ambient bed `-16`, music `-22`.
- `duck_db` (ambient/music; extra attenuation while dialogue plays; maps to a compressor ratio). Keep small (`~-5`) so beds stay audible under the voice.
- `fade_in`, `fade_out` (seconds, beds).
- optional `seed` (ambient/oneshot) for reproducibility, `gen_seconds` (oneshot gen length).
- optional `spatial` (used only by `mix_spatial.py`; ignored by `mix.py`/`generate_sfx.py`): stage position. One-shots take static `{pan,distance}` or movement `{from,to}`/`{path:[…]}`; ambient beds and score music take only `{pan}` (balance) and/or `{distance}` (fixed, no movement). Scene music takes `{scene:true, enter|exit:"left"|"right"|"front"}` or one `{from,to}` sweep — subtle, never zig-zag. See "Spatial mix internals".
`generate_sfx.py` writes back `file` (relative path) and `gen_duration`.
## Mixing internals (`mix.py`)
- Total timeline `T = max(dialogue, all cue ends)`.
- Voice formatted to stereo/48k, padded to `T`, `asplit` into a main copy + one sidechain key per **ambient** bed (music no longer takes a voice key — see below).
- `oneshot`: `volume=<gain>dB` -> `adelay=<start>` -> pad to `T`.
- `ambient`: ffmpeg `-stream_loop -1`, `atrim=0:<window>`, `afade` in/out, `volume`, `adelay`, pad; then `sidechaincompress` keyed off the **voice** (`ratio` derived from `duck_db`, `attack=20`, `release=350`).
- `music`: built the same way (loop/trim/fade/volume/pad) but **not ducked inline**. Instead all music beds duck under the whole **content bus** (voices + SFX) via `_common.assemble_content_and_music`. The duck is deliberately **smooth + shallow** (broadcast-style music ducking, not a fast pumping compressor): `MUSIC_DUCK_DB=-8`, `attack=40`, `release=1500` (rides through inter-word gaps, recovers only in real pauses), `threshold=0.04`, and the ratio is capped at `MUSIC_DUCK_RATIO_CAP=4`. So the ducking adds only a few dB of slow variation; keep the cue's base `gain_db` low (`-24..-20`) so even fully recovered the music never exceeds the dialogue. Per-cue `duck_db` overrides (still ratio-capped).
- Final assembly: the no-music content bus + the ducked music bus → `amix=normalize=0` → `loudnorm=I=-16:TP=-1.5:LRA=11` → `libmp3lame`.
- `--no-duck` disables ducking (music summed raw); `--target-i` changes loudness; `--bitrate` the MP3 rate.
- **Stems** (`--stems auto|always|off`, default `auto` = on when there are `music` cues): see "Music & stems" below.
## Spatial mix internals (`mix_spatial.py` + `spatial.py`)
`mix_spatial.py` is an opt-in, **theater-only** mixer that positions every source on a virtual stereo stage instead of the flat center mix of `mix.py`. It mixes the **per-line clips** (`lines.json`) like `split_tracks.py` (not the flattened `dialogue.wav`), so each voice can sit at its own position and move while talking. Output is a **normal stereo MP3** (works on speakers and headphones); the lipsync feed stays on `mix.py`.
**Stage model.** Each source has `pan ∈ [-1,+1]` (L..R, 0=center) and `distance ∈ [0,1]` (0=close, 1=far). Distance is three combined cues:
- **Level**: `dist_gain_dB = MAX_ATTEN_DB · distance` (`MAX_ATTEN_DB=-11`, override `--max-atten-db`).
- **Tone** (air absorption): `lowpass=f=lerp(18000 Hz near → 2600 Hz far)`.
- **Placement**: constant-power pan, `θ=(pan+1)/2·(π/2)`, `gL=cos θ`, `gR=sin θ` (mono fold-down stays level-stable).
**Static source** (no movement) — fast path, mono→stereo:
```
aformat=…channel_layouts=mono…, lowpass=f=<fc>, volume=<dist_gain>dB,
pan=stereo|c0=<gL>*c0|c1=<gR>*c0
```
**Moving source** — `aeval` is the only way to automate panning over time (`pan` is static). The mono input is duplicated to stereo first so `aeval` always has two input channels:
```
aformat=…mono…, pan=stereo|c0=c0|c1=c0, lowpass=f=<fc_mean>,
aeval=val(0)*<dg(t)>*cos(<ang(t)>)|val(1)*<dg(t)>*sin(<ang(t)>):channel_layout=stereo
```
- `ang(t)=(π/4)·(x(t)+1)`, `x(t)` = piecewise-linear pan path.
- `dg(t)=exp(0.11513·MAX_ATTEN_DB·d(t))` = linear distance gain (avoids `pow`, fewer commas).
- `d(t)` = piecewise-linear distance path. Both paths are built from keyframes with half-open `gte(t,a)*lt(t,b)` windows that sum cleanly, with flat clamping outside the range. `spatial.py` escapes the commas (`,`→`\,`) so the expression survives filtergraph parsing. Moving sources use a **single** low-pass at the mean-distance cutoff (cutoff can't be time-automated). `aeval` is per-sample/slow but the clips are short, so only the lines/cues that actually move pay the cost.
**Ambient beds stay stereo.** `ambient` beds are not collapsed/panned; a bed's optional `spatial.pan` applies a gentle **balance** (`pan=stereo|c0=gl*c0|c1=gr*c1`, keeps both channels and width) and optional `spatial.distance` adds level + low-pass. Beds keep the same loop/trim/fade/`sidechaincompress` ducking as `mix.py`; the **ambient-bed** duck key is the **summed positioned voices** (`asplit` of the voice mix into the main mix + one key per ambient bed). Music is handled separately (see "Spatial music").
**Narration priority.** So the narrator is never buried by a loud effect:
- **SFX distance floor** — `build_source_chain(min_distance=…)` (CLI `--sfx-min-distance`, default 0.12) clamps every SFX distance value (static and movement keyframes) to a minimum, so SFX render *behind* the dialogue plane while the narrator (≈0.05) stays the closest source. Voices pass `min_distance=0`.
- **SFX duck under narration** — one-shots are `sidechaincompress`-ed (faster `attack=5`, `release=250`) keyed off a **narration-only bus**, built by `asplit`-ing each narration voice into the voice mix *and* a key copy, then `amix`-ing the key copies into `[narrbus]` (further `asplit` into one key per one-shot). Keying off the narrator only (not all voices) means SFX stay punchy under character dialogue and in gaps, but always dip under the narrator. `--sfx-duck-db` (default -6) sets the amount; if there's no narration character, SFX key off the full voice mix; `--no-duck-sfx` skips it; `--no-duck` skips all ducking.
**Intelligibility.** Voice pan is clamped to `±--voice-pan-limit` (default 0.5) so dialogue never hard-pans; SFX go to `±--sfx-pan-limit` (default 0.95). Narration auto-seats center/intimate; other characters auto-seat at alternating L/R offsets when no `stage` is given. `--crossfeed` adds headphone crossfeed (softens hard pans); `--target-i`, `--bitrate`, `--output-name` behave as in `mix.py`.
**Authoring for *audible* motion** (all optional, backward compatible): character `stage` + per-line `spatial` in `script.json`, per-cue `spatial` in `cues.json`. Shapes: static `{pan,distance}`, `{from:{…},to:{…}}`, `{path:[{t,pan,distance},…]}` (`t` seconds from source start). Subtle moves don't read against beds + dialogue — **use wide pans and big distance swings**: an approaching source (distance ↓, gets louder/brighter) or a receding one (distance ↑, gets quieter/darker) is the strongest cue, ideally combined with an L→R pan. See `historia-breve` for a worked demo: Lila's voice enters from far-left and walks to center-near (line 6), the owl flies L→R *and recedes* (path with rising distance), the footsteps approach from far-left, the gust sweeps past and away, the fire bed is balanced to Mateo's side.
**Binaural (future).** True 3D with front/back needs HRTF (`sofalizer`/`libmysofa`), not compiled into the current ffmpeg. Deferred — would require recompiling ffmpeg or a NumPy + SOFA render pass. Stereo spatialization was chosen for universal compatibility and zero new dependencies.
**Spatial music.** A `music` cue is a **fixed wide stereo bed by default** (non-diegetic score / narrator underscore — no movement). It becomes a **positioned point source** (diegetic scene music: a music box, a radio) only with `spatial.scene=true`. Scene motion is deliberately **subtle and single-direction** — `music_scene_spatial()` sanitizes the cue's `spatial` before `build_source_chain`:
- `enter: "left"|"right"|"front"` → a 3-keyframe path that glides in from that side/depth over `min(5s, win/2)`, then **holds** at the seat (no zig-zag).
- `exit: "left"|"right"|"front"` → holds, then glides out over the last few seconds.
- `from`/`to` → one sweep; a multi-point `path` is collapsed to first→last.
Whether wide bed or scene point source, music is **not** ducked inline here — it joins the music bus and ducks under the content (voices + SFX) in `assemble_content_and_music`, same as `mix.py`. Scene cues are reported as `scene_music: [ids]`; `music_ducks_under: "content (voices+sfx)"` in the summary.
## Music & stems
`music` cues are **always instrumental** and generated by `generate_sfx.py` (`gen_music`) via `--music-backend`:
- `hq` → **MiniMax Music 2.6** with `is_instrumental=true` (`audio_music.py`): the model ignores `lyrics` and never sings; the `prompt` (built from `description` + `mood`, reading `bg-music-hq`'s `moods.json` read-only) drives the track. Default; best for scores, podcast beds, scene music. Uses the predictions API + polling (not `replicate.run`, whose short sync wait times out on 1-3 min jobs).
- `fast` → `bg-music`: quicker/lighter instrumental.
- `auto` → `hq`. Per-cue `music_backend` overrides; cue `mood` + window length are passed through. The model returns a full-length track; if it's shorter than the cue window the mixer stream-loops it.
> `bg-music-hq` is **not** used for theater music: it targets `music-2.5` (no instrumental flag) and feeds structure directions through `lyrics`, which `music-2.5` sings literally. `audio_music.py` only *reads* its mood library; it never modifies that skill.
**Three tracks.** When a project has `music` cues, both `mix.py` and `mix_spatial.py` emit three MP3s (derived from `--output-name`, e.g. `final.mp3` → `final.nomusic.mp3`, `final.music.mp3`):
- `final.mp3` — full mix (dialogue + SFX + music).
- `final.nomusic.mp3` — dialogue + SFX, music excluded (→ `seedance-2`).
- `final.music.mp3` — music only, **with the exact ducking/fades/spatial it has in the full mix**.
**How (shared linear gain → exact recombination).** `_common.assemble_content_and_music` sums the **content bus** (`[voice]` + all non-`music` cue labels) into `nmraw`, `asplit`s it into a keep copy + one sidechain key per music bed, ducks each music bed under its key, and sums the ducked beds into the music bus (crossfeed applied per bus in `mix_spatial.py` — it's linear). The mixer renders the two buses as pre-norm WAV partitions in one ffmpeg pass (content bus → `nomusic`, ducked music bus → `music`); the partitions share **no signal**, so `full == nomusic + music`. `_common.finalize_stems` then:
1. `amix(nomusicpre, musicpre, normalize=0)` → `full_pre.wav`.
2. `shared_norm_gain_db(full_pre)` = `min(target_I − input_I, target_TP − input_TP)` from a `loudnorm` analysis pass — integrated loudness shifts 1:1 with a fixed gain, capped so the true peak stays ≤ target.
3. Apply that **same** `volume=<gain>dB` to all three WAVs → MP3s.
Because the gain is a single linear factor applied to a partition, `final == nomusic + music` sample-for-sample (pre-encode); the only residual after independent MP3 encoding is the lossy noise floor (measured ≈ −52 to −59 dB, i.e. ~30 dB below signal). `mix_spatial.py` applies `crossfeed` per stem before finalizing — crossfeed is linear, so the recombination still holds. Verify with:
```
ffmpeg -i final.mp3 -i final.nomusic.mp3 -i final.music.mp3 -filter_complex \
"[1][2]amix=inputs=2:normalize=0[s];[s]pan=stereo|c0=-1*c0|c1=-1*c1[n];[0][n]amix=inputs=2:normalize=0[d];[d]astats[o]" \
-map "[o]" -f null - # residual RMS should be ≈ MP3 noise floor
```
**Storyboard overlay.** Feed `final.nomusic.mp3` (music-free) to `seedance-2` so the model isn't confused by music, then overlay `final.music.mp3` over the returned video (it keeps the original timecodes + ducking, so it lines up and dips under dialogue automatically):
```
ffmpeg -i reel.mp4 -i final.music.mp3 -filter_complex \
"[0:a][1:a]amix=inputs=2:normalize=0:dropout_transition=0,loudnorm=I=-16:TP=-1.5:LRA=11[a]" \
-map 0:v -map "[a]" -c:v copy -c:a aac -shortest reel_final.mp4
```
## Lipsync handoff to seedance-2
`export_lipsync.py` validates each clip `<=15s` (`ok`) and emits `lipsync.json`. For each clip:
1. `media_upload` (Higgsfield MCP) -> PUT the clip bytes to `upload_url` -> `media_confirm` (`type: "audio"`).
2. Build the prompt with the exact transcript:
```bash
python3 ~/.cursor/skills/seedance-2/scripts/prompt_tools.py reframe \
"[Image1] the character speaks to camera, natural lip movement" \
--images 1 --audios 1 --audio-transcript "<clip transcript>"
```
3. `generate_video` (model `seedance_2_0`) with the storyboard panel image (reference-image role) + the confirmed audio `media_id` (audio role). Seedance needs at least one image/video reference and total `<=15s`. Map `clips[*].panel` to your storyboard panels.
The skill does not generate the storyboard — make it with `gpt-image-2` / `seedance-2`.
**Audio must stay strictly *under* the render duration.** Seedance fails or mistimes when audio length ≥ requested duration. For a 15s render, trim the feed to ≤14.9s (small tail fade); the result comes back ~15s.
## split_tracks.py — narration vs on-camera (for video lip-sync)
`split_tracks.py` rebuilds two role-separated tracks from the clean per-line clips + cues, so an audio-driven video model lip-syncs only the on-camera character and never the off-camera narrator.
```bash
python3 $SCRIPTS/split_tracks.py -o $OUT
python3 $SCRIPTS/split_tracks.py -o $OUT --narration "Narrador" --onscreen "Doki" # override
```
- **Role detection**: a character is narration when `script.json` gives it `role` ∈ {narration, narrator, voiceover, vo, offscreen, off_camera} or `on_camera: false`; everything else is on-camera. `--narration`/`--onscreen` (comma lists) override.
- **Outputs** (into `$OUT`):
- `narration.mp3` — narration lines only, placed at their original timecodes (silence elsewhere), loudnorm'd.
- `lipsync_mix.mp3` — on-camera voices placed on a silent bed + all SFX (via `mix.py`, same ducking), **narration muted**. Feed this to the video model.
- `tracks.json` — per-line track assignment + durations.
- **Internals**: each clean clip is placed with `adelay` on a full-length `apad` bed and summed (`amix=normalize=0`); the on-camera bed is handed to `mix.py` as `--dialogue` with `--output-name lipsync_mix.mp3` so SFX/ducking match the theater mix exactly.
- **Post-merge**: after the video renders, mix `narration.mp3` back onto the video's audio (it already carries the on-camera voice + SFX). Original timecodes mean it lines up with the visuals automatically:
```bash
ffmpeg -y -i reel.mp4 -i narration.mp3 \
-filter_complex "[0:a][1:a]amix=inputs=2:normalize=0:dropout_transition=0,loudnorm=I=-16:TP=-1.5:LRA=11[a]" \
-map 0:v -map "[a]" -c:v copy -c:a aac -shortest reel_final.mp4
```
## Recipes
### Radio play (theater)
```bash
SCRIPTS=~/.cursor/skills/audio-theater/scripts; OUT=audio-theater/tormenta
python3 $SCRIPTS/write_script.py --idea "Dos marineros y una tormenta" --mode theater -l es -o $OUT
python3 $SCRIPTS/generate_voices.py --script $OUT/script.json -o $OUT
python3 $SCRIPTS/transcribe.py --script $OUT/script.json -o $OUT
# author $OUT/cues.json (ambient rain bed + door slam one-shot)
python3 $SCRIPTS/generate_sfx.py -o $OUT
python3 $SCRIPTS/mix.py -o $OUT
python3 $SCRIPTS/build_transcript.py -o $OUT
```
### Lipsync clips for seedance-2
```bash
python3 $SCRIPTS/write_script.py --idea "..." --mode lipsync -l es -o $OUT
python3 $SCRIPTS/generate_voices.py --script $OUT/script.json -o $OUT # keep lines short
python3 $SCRIPTS/export_lipsync.py -o $OUT
# -> $OUT/lipsync.json + clean $OUT/lines/*.wav, then drive seedance-2 per clip
```
### Podcast (mascotify)
```bash
python3 $SCRIPTS/write_script.py --idea "Por que los gatos amasan" --mode podcast \
-l es --hosts "Lucas,Mia" -o $OUT
python3 $SCRIPTS/generate_voices.py --script $OUT/script.json -o $OUT --two-speaker
python3 $SCRIPTS/transcribe.py -o $OUT
# author cues.json: intro_music (mood podcast-intro), optional podcast-bed, outro_music
python3 $SCRIPTS/generate_sfx.py -o $OUT
python3 $SCRIPTS/mix.py -o $OUT
python3 $SCRIPTS/build_transcript.py -o $OUT # show-notes
```
## Troubleshooting
- **No Gemini key / Replicate token**: run `setup_key.py --show`; ensure asset-generator (Gemini) and sound-effects (Replicate) are configured, or set env vars / `--gemini` / `--replicate`.
- **TTS reads instructions aloud or returns text**: it retries automatically; if persistent, simplify the line, ensure the language matches the chosen voice's natural tone, or try `--model gemini-2.5-pro-preview-tts`.
- **Clip too long for lipsync**: split the offending line in `script.json` and re-run `generate_voices.py` + `export_lipsync.py`.
- **faster-whisper missing**: `pip3 install faster-whisper` (heavy: torch/ctranslate2), or use `--backend replicate`.
- **ffmpeg filter error in mix**: the failing `filter_complex` is printed to stderr; check cue `start`/`end` values are numeric and `end > start`.
- **Looping bed sounds repetitive**: increase the generated length (longer `end-start`) or pick a richer `description`.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |