Skip to content

Diogenesoftoronto/keating / video/keating-surface-tour/.agents/skills/media-use

media-use

Flagged

Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.

FileKindSize
.gitignoreother17 B
SKILL.mdskill md7.8 KB
audio/assets/sfx/CREDITS.mdmarkdown1.2 KB
audio/assets/sfx/chime.mp3binary26.7 KB
audio/assets/sfx/click-soft.mp3binary11.4 KB
audio/assets/sfx/click.mp3binary11.4 KB
audio/assets/sfx/error.mp3binary50.6 KB
audio/assets/sfx/glitch-1.mp3binary82.4 KB
audio/assets/sfx/glitch-2.mp3binary109.5 KB
audio/assets/sfx/glitch-3.mp3binary96.8 KB
audio/assets/sfx/impact-bass-1.mp3binary66.1 KB
audio/assets/sfx/impact-bass-2.mp3binary81.0 KB
audio/assets/sfx/key-press.mp3binary3.8 KB
audio/assets/sfx/manifest.jsonconfig3.6 KB
audio/assets/sfx/notification.mp3binary76.7 KB
audio/assets/sfx/ping.mp3binary25.8 KB
audio/assets/sfx/pop.mp3binary22.5 KB
audio/assets/sfx/riser.mp3binary313.5 KB
audio/assets/sfx/sparkle.mp3binary56.3 KB
audio/assets/sfx/typing.mp3binary26.2 KB
audio/assets/sfx/whoosh-cinematic.mp3binary173.3 KB
audio/assets/sfx/whoosh-short.mp3binary18.0 KB
audio/assets/sfx/whoosh.mp3binary18.0 KB
audio/references/bgm.mdmarkdown7.8 KB
audio/references/captions/authoring.mdmarkdown8.8 KB
audio/references/captions/motion.mdmarkdown5.5 KB
audio/references/captions/transcript-handling.mdmarkdown5.3 KB
audio/references/remove-background.mdmarkdown8.3 KB
audio/references/requirements.mdmarkdown4.2 KB
audio/references/sfx.mdmarkdown3.4 KB
audio/references/transcribe.mdmarkdown2.7 KB
audio/references/tts-to-captions.mdmarkdown1.2 KB
audio/references/tts.mdmarkdown8.6 KB
audio/scripts/audio.mjsscript12.5 KB
audio/scripts/audio.test.mjsscript1.9 KB
audio/scripts/heygen-tts.mjsexecutablescript4.2 KB
audio/scripts/lib/bgm.mjsscript11.3 KB
audio/scripts/lib/bgm.test.mjsscript1.2 KB
audio/scripts/lib/concurrency.mjsscript527 B
audio/scripts/lib/concurrency.test.mjsscript1.6 KB
audio/scripts/lib/heygen.mjsscript6.9 KB
audio/scripts/lib/heygen.test.mjsscript3.5 KB
audio/scripts/lib/python.mjsscript2.5 KB
audio/scripts/lib/python.test.mjsscript2.6 KB
audio/scripts/lib/sfx.mjsscript5.3 KB
audio/scripts/lib/sfx.test.mjsscript2.8 KB
audio/scripts/lib/tts.mjsscript16.1 KB
audio/scripts/lib/tts.spawn.test.mjsscript5.1 KB
audio/scripts/lib/tts.test.mjsscript6.0 KB
audio/scripts/lyria-recipe.pyscript4.4 KB
audio/scripts/wait-bgm.mjsscript5.4 KB
luts/README.mdmarkdown1.1 KB
luts/index.jsonconfig2.0 KB
references/audio.mdmarkdown1.7 KB
references/grading.mdmarkdown6.6 KB
references/media-treatment-recipes.mdmarkdown35.0 KB
references/media-treatments.mdmarkdown13.6 KB
references/memory.mdmarkdown3.3 KB
references/meta.mdmarkdown6.6 KB
references/operations.mdmarkdown14.1 KB
references/resolve.mdmarkdown8.8 KB
references/setup-providers.mdmarkdown7.1 KB
references/telemetry-dashboard.mdmarkdown4.1 KB
scripts/audio-duck.mjsscript3.9 KB
scripts/dither.mjsscript9.7 KB
scripts/dither.test.mjsscript4.6 KB
scripts/eval.mjsscript14.5 KB
scripts/lib/adopt.mjsscript4.3 KB
scripts/lib/adopt.test.mjsscript2.7 KB
scripts/lib/bgm-provider.mjsscript650 B
scripts/lib/brand-provider.mjsscript1.6 KB
scripts/lib/bundled-sfx-provider.mjsscript3.5 KB
scripts/lib/bundled-sfx-provider.test.mjsscript3.1 KB
scripts/lib/cache.mjsscript4.8 KB
scripts/lib/candidates.mjsscript4.0 KB
scripts/lib/candidates.test.mjsscript5.9 KB
scripts/lib/codex-provider.mjsscript6.5 KB
scripts/lib/codex-provider.test.mjsscript1.3 KB
scripts/lib/coverage.test.mjsscript5.6 KB
scripts/lib/cube-build.mjsscript5.7 KB
scripts/lib/cube-build.test.mjsscript2.7 KB
scripts/lib/cube-validate.mjsscript5.9 KB
scripts/lib/cube-validate.test.mjsscript3.2 KB
scripts/lib/cutlist.mjsscript6.0 KB
scripts/lib/cutlist.test.mjsscript4.0 KB
scripts/lib/duck.mjsscript3.0 KB
scripts/lib/duck.test.mjsscript3.4 KB
scripts/lib/error-diffusion.mjsscript6.5 KB
scripts/lib/error-diffusion.test.mjsscript3.4 KB
scripts/lib/freeze.mjsscript3.0 KB
scripts/lib/freeze.test.mjsscript1.9 KB
scripts/lib/grade-analyzer.mjsscript8.6 KB
scripts/lib/grade-analyzer.test.mjsscript7.7 KB
scripts/lib/heygen-cli.mjsscript7.3 KB
scripts/lib/heygen-cli.test.mjsscript10.0 KB
scripts/lib/heygen-search.mjsscript1.8 KB
scripts/lib/heygen-search.test.mjsscript1.8 KB
scripts/lib/heygen-video-provider.mjsscript3.6 KB
scripts/lib/heygen-video-provider.test.mjsscript12.0 KB
scripts/lib/image-provider.mjsscript1.4 KB
scripts/lib/index-gen.mjsscript1.9 KB
scripts/lib/local-models.mjsscript13.3 KB
scripts/lib/local-models.test.mjsscript5.6 KB
scripts/lib/local-run.mjsscript2.2 KB
scripts/lib/local-run.test.mjsscript1.7 KB
scripts/lib/logo-provider.mjsscript7.0 KB
scripts/lib/logo-provider.test.mjsscript5.7 KB
scripts/lib/ltx-video-provider.mjsscript1.9 KB
scripts/lib/ltx-video-provider.test.mjsscript3.8 KB
scripts/lib/lut-preset-provider.mjsscript8.5 KB
scripts/lib/lut-preset-provider.test.mjsscript6.3 KB
scripts/lib/manifest.mjsscript7.0 KB
scripts/lib/manifest.test.mjsscript11.6 KB
scripts/lib/match.mjsscript1.4 KB
scripts/lib/mflux-provider.mjsscript3.2 KB
scripts/lib/misses.mjsscript1.2 KB
scripts/lib/misses.test.mjsscript2.5 KB
scripts/lib/npx-sync.mjsscript1.3 KB
scripts/lib/npx-sync.test.mjsscript2.0 KB
scripts/lib/parakeet-words.mjsscript1.1 KB
scripts/lib/parakeet-words.test.mjsscript1.4 KB
scripts/lib/prefs-store.mjsscript6.4 KB
scripts/lib/prefs-store.test.mjsscript7.2 KB
scripts/lib/probe.mjsscript1.3 KB
scripts/lib/probe.test.mjsscript1.6 KB
scripts/lib/providers.mjsscript327 B
scripts/lib/recipe-store.mjsscript12.7 KB
scripts/lib/recipe-store.test.mjsscript12.5 KB
scripts/lib/registry.mjsscript9.5 KB
scripts/lib/registry.test.mjsscript9.6 KB
scripts/lib/search.mjsscript906 B
scripts/lib/search.test.mjsscript1.2 KB
scripts/lib/sfx-provider.mjsscript700 B
scripts/lib/specs.mjsscript3.2 KB
scripts/lib/specs.test.mjsscript2.7 KB
scripts/lib/stats.mjsscript3.7 KB
scripts/lib/stats.test.mjsscript5.4 KB
scripts/lib/telemetry.mjsscript8.3 KB
scripts/lib/telemetry.test.mjsscript13.1 KB
scripts/lib/tts-local-provider.mjsscript3.7 KB
scripts/lib/tts-local-provider.test.mjsscript2.6 KB
scripts/lib/usage.mjsscript1.8 KB
scripts/lib/usage.test.mjsscript1.7 KB
scripts/lib/voice-provider.mjsscript1.7 KB
scripts/lib/voice-provider.test.mjsscript1.9 KB
scripts/lib/words.mjsscript764 B
scripts/prefs.mjsscript2.4 KB
scripts/recipe.mjsscript4.0 KB
scripts/resolve.mjsscript43.2 KB
scripts/resolve.test.mjsscript35.8 KB
scripts/transcribe.mjsscript6.5 KB
scripts/transcript-cut.mjsscript7.1 KB

audio/scripts/audio.mjs

View on GitHub
#!/usr/bin/env node
// audio.mjs — the shared HyperFrames audio engine. ONE implementation of TTS +
// BGM + SFX for every video workflow (product-launch, general-video, pr-to-video,
// …). Workflows do NOT vendor a copy: they write a neutral `audio_request.json`
// (a tiny per-workflow adapter maps their storyboard/scenes into it) and call:
//
//   node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json
//
// The three capabilities degrade on ONE switch — whether HeyGen is configured
// (credential present, NOT the CLI). This mirrors the table in ../SKILL.md:
//
//   TTS : HeyGen REST → ElevenLabs → Kokoro (CLI)
//   BGM : HeyGen retrieve  → (no credential) Lyria/MusicGen generate
//   SFX : HeyGen retrieve  → (no credential) bundled 19-file library
//
// ── audio_request.json (input) ────────────────────────────────────────────────
//   {
//     "provider": "auto",          // auto|heygen|elevenlabs|kokoro (override: --provider)
//     "lang": "en", "speed": 1.0,
//     "lines": [                   // one TTS unit each; id joins back to the caller's model
//       { "id": "01", "text": "...", "sfx": ["whoosh", "ui click"] }
//     ],
//     "bgm": { "mode": "retrieve", // retrieve|generate|none (override: --bgm-mode / --no-bgm)
//              "query": "calm cinematic underscore",   // mood for retrieval
//              "prompt": null,      // full prompt for generation (else inferred)
//              "blob": "...", "archetype": "...", "arc": "..." }  // optional mood-inference hints
//   }
//
// ── audio_meta.json (output, id-keyed) ───────────────────────────────────────
//   { tts_provider, voice_id,
//     bgm: { path, volume, mode, query?, duration_s? } | null,
//     bgm_pending, bgm_provider, bgm_pid, bgm_log, bgm_mode, bgm_target_duration_s, …,
//     voices: [ { id, path, duration_s, words: [{id,text,start,end}] } ],
//     sfx:    [ { id, name, file, source, offset_s, duration_s, volume } ],
//     total_duration_s }
//
// --only tts,bgm,sfx  runs a subset and MERGES into an existing --out (so a
// workflow can do TTS+BGM early, then SFX later once cues exist). When BGM uses
// the generate path it is spawned detached (bgm_pending:true) — run wait-bgm.mjs
// before assembling.

import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { heygenAuthHeaders, heygenCredential, loadEnvFromDir } from "./lib/heygen.mjs";
import {
  ffprobeDuration,
  pickProvider,
  resolveVoiceId,
  synthesizeOne,
  transcribeWav,
  withWordIds,
} from "./lib/tts.mjs";
import { generateBgmDetached, inferBgmPrompt, retrieveBgm } from "./lib/bgm.mjs";
import { resolveSfx } from "./lib/sfx.mjs";
import { mapWithConcurrency } from "./lib/concurrency.mjs";

const HERE = dirname(fileURLToPath(import.meta.url));
const argv = process.argv.slice(2);
const flag = (name, def) => {
  const i = argv.indexOf(`--${name}`);
  return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
};
const has = (name) => argv.includes(`--${name}`);
const die = (m) => {
  console.error(`✗ audio engine: ${m}`);
  process.exit(1);
};
const r3 = (x) => Number(x.toFixed(3));

// Two independent reports of an unbounded Promise.all over TTS lines
// overwhelming a machine: one OOM'd 12/13 concurrent Kokoro TTS +
// whisper-transcribe lines on a resource-constrained laptop, the other saw
// 7/8 lines fail on first run (concurrent cold-start model loads) and pass on
// retry once the model was cached. Kokoro/Whisper each load their own local
// model per subprocess, so firing every line at once multiplies that cost by
// the line count. mapWithConcurrency caps how many run at once — still
// parallel, just bounded.
const ttsConcurrency = Math.max(1, Number(process.env.HYPERFRAMES_TTS_CONCURRENCY) || 4);

const hyperframesDir = resolve(flag("hyperframes", "."));
const requestPath = resolve(flag("request", join(hyperframesDir, "audio_request.json")));
const outPath = resolve(flag("out", join(hyperframesDir, "audio_meta.json")));
const sfxLibDir = resolve(flag("sfx-lib", join(HERE, "..", "assets", "sfx")));
const lyriaRecipe = resolve(flag("lyria-recipe", join(HERE, "lyria-recipe.py")));
const onlyArg = flag("only", "tts,bgm,sfx");
const only = new Set(
  onlyArg
    .split(",")
    .map((s) => s.trim())
    .filter(Boolean),
);
const providerOverride = flag("provider", null);
const bgmModeOverride = flag("bgm-mode", null);
const noBgm = has("no-bgm");
const voiceOverride = flag("voice", null);
const speedOverride = flag("speed", null);
const langOverride = flag("lang", null);
const seedSeconds = Number(flag("seed-seconds", "28")) || 28;

if (!existsSync(requestPath)) die(`audio_request.json not found at ${requestPath}`);
let request;
try {
  request = JSON.parse(readFileSync(requestPath, "utf8"));
} catch (e) {
  die(`audio_request.json parse: ${e.message}`);
}
const lines = Array.isArray(request.lines) ? request.lines : [];
const lang = langOverride || request.lang || "en";
const speed = Number(speedOverride ?? request.speed ?? 1.0) || 1.0;

// ── env + HeyGen availability (the single switch) ─────────────────────────────
loadEnvFromDir(hyperframesDir);
const heygenOK = heygenCredential() !== null;
const headers = heygenOK ? heygenAuthHeaders() : null;

// ── merge base: preserve sections not selected by --only ──────────────────────
const prev = existsSync(outPath) ? JSON.parse(readFileSync(outPath, "utf8")) : {};
const anomalies = [];

// ── TTS ───────────────────────────────────────────────────────────────────────
let voices = prev.voices ?? [];
let ttsProvider = prev.tts_provider ?? null;
let voiceId = prev.voice_id ?? null;
if (only.has("tts") && lines.length) {
  try {
    ttsProvider = pickProvider(
      providerOverride || (request.provider === "auto" ? null : request.provider),
    );
  } catch (e) {
    die(e.message);
  }
  voiceId = await resolveVoiceId({
    provider: ttsProvider,
    userVoice: voiceOverride || request.voice,
    lang,
  });
  console.error(`· tts: ${ttsProvider} · voice ${voiceId} · ${lines.length} line(s)`);
  const synthLine = async (line) => {
    const id = String(line.id);
    const text = String(line.text ?? "").trim();
    if (!text) {
      anomalies.push(`line ${id}: empty text — skipped`);
      return null;
    }
    const rel = `assets/voice/${id}.wav`;
    const abs = join(hyperframesDir, rel);
    const { ok, words, error } = await synthesizeOne({
      provider: ttsProvider,
      text,
      voiceId,
      lang,
      speed,
      wavAbs: abs,
      hyperframesDir,
    });
    if (!ok) {
      anomalies.push(`line ${id}: TTS failed — omitted${error ? ` (${error})` : ""}`);
      return null;
    }
    let wordArr = words; // heygen: native; else transcribe
    if (!wordArr) wordArr = await transcribeWav({ wavRel: rel, lang, hyperframesDir });
    const dur = ffprobeDuration(abs);
    if (!isFinite(dur) || dur <= 0) {
      anomalies.push(`line ${id}: bad voice duration — omitted`);
      return null;
    }
    return { id, path: rel, duration_s: r3(dur), words: withWordIds(wordArr) };
  };
  const results = await mapWithConcurrency(lines, ttsConcurrency, synthLine);
  voices = results.filter(Boolean);
  for (const v of voices)
    console.error(`  voice ${v.id}: ${v.path} (${v.duration_s}s, ${v.words.length} words)`);
}
const hasVoice = voices.length > 0;
const totalDuration = r3(voices.reduce((a, v) => a + (v.duration_s || 0), 0));

// ── BGM ─────────────────────────────────────────────────────────────────────
let bgm = prev.bgm ?? null;
const bgmFields = {
  bgm_pending: prev.bgm_pending ?? false,
  bgm_provider: prev.bgm_provider ?? null,
  bgm_pid: prev.bgm_pid ?? null,
  bgm_log: prev.bgm_log ?? null,
  bgm_mode: prev.bgm_mode ?? null,
  bgm_target_duration_s: prev.bgm_target_duration_s ?? null,
  bgm_seed_duration_s: prev.bgm_seed_duration_s ?? null,
  bgm_loop_count: prev.bgm_loop_count ?? null,
};
if (only.has("bgm")) {
  bgm = null;
  Object.keys(bgmFields).forEach((k) => (bgmFields[k] = k === "bgm_pending" ? false : null));
  // Mode resolution. An EXPLICIT mode (flag or request.bgm.mode) is strict:
  // "retrieve" means retrieve-or-nothing — it never silently starts a detached
  // generate (a caller with no wait-bgm step, e.g. product-launch, must not get
  // a pending job it can't await). Only the UNSET/auto default picks generate
  // when HeyGen is absent.
  const explicitMode = bgmModeOverride || request.bgm?.mode || null;
  let mode = noBgm ? "none" : explicitMode || (heygenOK ? "retrieve" : "generate");
  if (mode === "retrieve" && !heygenOK) {
    anomalies.push(
      "bgm: retrieve requires a HeyGen credential — skipped (no generate fallback for an explicit retrieve)",
    );
    mode = "none";
  }

  if (mode === "none") {
    console.error(`· bgm: disabled`);
  } else if (mode === "retrieve") {
    try {
      bgm = await retrieveBgm({ query: request.bgm?.query, headers, hyperframesDir, hasVoice });
      if (bgm) {
        bgmFields.bgm_provider = "heygen";
        bgmFields.bgm_mode = "retrieve";
        console.error(`  bgm: ${bgm.path} (retrieve "${bgm.query}")`);
      } else {
        anomalies.push(`bgm: no music match for "${request.bgm?.query ?? ""}" — skipped`);
      }
    } catch (e) {
      anomalies.push(`bgm retrieve failed: ${e.message} — skipped`);
    }
  } else {
    // generate
    const prompt = inferBgmPrompt({
      userPrompt: request.bgm?.prompt,
      blob: request.bgm?.blob || request.bgm?.query,
      archetype: request.bgm?.archetype,
      arc: request.bgm?.arc,
    });
    const gen = generateBgmDetached({
      prompt,
      durationS: totalDuration || 30,
      hyperframesDir,
      lyriaRecipe: existsSync(lyriaRecipe) ? lyriaRecipe : null,
      seedSeconds,
      hasVoice,
    });
    if (gen.disabled) {
      anomalies.push(`bgm: ${gen.reason}`);
    } else {
      bgm = { path: gen.path, volume: gen.volume, mode: gen.mode, duration_s: null };
      bgmFields.bgm_pending = true;
      bgmFields.bgm_provider = gen.provider;
      bgmFields.bgm_pid = gen.pid;
      bgmFields.bgm_log = gen.log;
      bgmFields.bgm_mode = gen.mode;
      bgmFields.bgm_target_duration_s = gen.target_duration_s ?? null;
      bgmFields.bgm_seed_duration_s = gen.seed_duration_s ?? null;
      bgmFields.bgm_loop_count = gen.loop_count ?? null;
      console.error(`  bgm: launched ${gen.provider} (detached, pid ${gen.pid}) → ${gen.path}`);
    }
  }
}

// ── SFX ─────────────────────────────────────────────────────────────────────
let sfx = prev.sfx ?? [];
if (only.has("sfx")) {
  const cues = lines.flatMap((l) =>
    (Array.isArray(l.sfx) ? l.sfx : [])
      .map((name) => ({ id: String(l.id), name: String(name).trim() }))
      .filter((c) => c.name),
  );
  const res = await resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir });
  sfx = res.sfx;
  anomalies.push(...res.anomalies);
  console.error(
    `· sfx: ${sfx.length} cue(s) resolved (${heygenOK ? "heygen retrieval" : "bundled library"})`,
  );
}

// ── write audio_meta.json ─────────────────────────────────────────────────────
const meta = {
  tts_provider: ttsProvider,
  voice_id: voiceId,
  bgm,
  ...bgmFields,
  voices,
  sfx,
  total_duration_s: totalDuration,
};
mkdirSync(dirname(outPath), { recursive: true });
writeFileSync(outPath, JSON.stringify(meta, null, 2));

console.log(`✓ audio engine → ${outPath}`);
console.log(`  heygen: ${heygenOK ? "yes" : "no"}  ·  ran: ${[...only].join(",")}`);
console.log(
  `  voices: ${voices.length}  ·  bgm: ${bgm ? `${bgmFields.bgm_provider}${bgmFields.bgm_pending ? " (pending)" : ""}` : "none"}  ·  sfx: ${sfx.length}`,
);
console.log(`  total voice duration: ${totalDuration}s`);
if (anomalies.length) {
  console.log(`\nanomalies (non-fatal):`);
  for (const a of anomalies) console.log(`  - ${a}`);
}
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall