Skip to content

Diogenesoftoronto/keating / video/keating-surface-tour/.agents/skills/media-use

media-use

Flagged

Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.

FileKindSize
.gitignoreother17 B
SKILL.mdskill md7.8 KB
audio/assets/sfx/CREDITS.mdmarkdown1.2 KB
audio/assets/sfx/chime.mp3binary26.7 KB
audio/assets/sfx/click-soft.mp3binary11.4 KB
audio/assets/sfx/click.mp3binary11.4 KB
audio/assets/sfx/error.mp3binary50.6 KB
audio/assets/sfx/glitch-1.mp3binary82.4 KB
audio/assets/sfx/glitch-2.mp3binary109.5 KB
audio/assets/sfx/glitch-3.mp3binary96.8 KB
audio/assets/sfx/impact-bass-1.mp3binary66.1 KB
audio/assets/sfx/impact-bass-2.mp3binary81.0 KB
audio/assets/sfx/key-press.mp3binary3.8 KB
audio/assets/sfx/manifest.jsonconfig3.6 KB
audio/assets/sfx/notification.mp3binary76.7 KB
audio/assets/sfx/ping.mp3binary25.8 KB
audio/assets/sfx/pop.mp3binary22.5 KB
audio/assets/sfx/riser.mp3binary313.5 KB
audio/assets/sfx/sparkle.mp3binary56.3 KB
audio/assets/sfx/typing.mp3binary26.2 KB
audio/assets/sfx/whoosh-cinematic.mp3binary173.3 KB
audio/assets/sfx/whoosh-short.mp3binary18.0 KB
audio/assets/sfx/whoosh.mp3binary18.0 KB
audio/references/bgm.mdmarkdown7.8 KB
audio/references/captions/authoring.mdmarkdown8.8 KB
audio/references/captions/motion.mdmarkdown5.5 KB
audio/references/captions/transcript-handling.mdmarkdown5.3 KB
audio/references/remove-background.mdmarkdown8.3 KB
audio/references/requirements.mdmarkdown4.2 KB
audio/references/sfx.mdmarkdown3.4 KB
audio/references/transcribe.mdmarkdown2.7 KB
audio/references/tts-to-captions.mdmarkdown1.2 KB
audio/references/tts.mdmarkdown8.6 KB
audio/scripts/audio.mjsscript12.5 KB
audio/scripts/audio.test.mjsscript1.9 KB
audio/scripts/heygen-tts.mjsexecutablescript4.2 KB
audio/scripts/lib/bgm.mjsscript11.3 KB
audio/scripts/lib/bgm.test.mjsscript1.2 KB
audio/scripts/lib/concurrency.mjsscript527 B
audio/scripts/lib/concurrency.test.mjsscript1.6 KB
audio/scripts/lib/heygen.mjsscript6.9 KB
audio/scripts/lib/heygen.test.mjsscript3.5 KB
audio/scripts/lib/python.mjsscript2.5 KB
audio/scripts/lib/python.test.mjsscript2.6 KB
audio/scripts/lib/sfx.mjsscript5.3 KB
audio/scripts/lib/sfx.test.mjsscript2.8 KB
audio/scripts/lib/tts.mjsscript16.1 KB
audio/scripts/lib/tts.spawn.test.mjsscript5.1 KB
audio/scripts/lib/tts.test.mjsscript6.0 KB
audio/scripts/lyria-recipe.pyscript4.4 KB
audio/scripts/wait-bgm.mjsscript5.4 KB
luts/README.mdmarkdown1.1 KB
luts/index.jsonconfig2.0 KB
references/audio.mdmarkdown1.7 KB
references/grading.mdmarkdown6.6 KB
references/media-treatment-recipes.mdmarkdown35.0 KB
references/media-treatments.mdmarkdown13.6 KB
references/memory.mdmarkdown3.3 KB
references/meta.mdmarkdown6.6 KB
references/operations.mdmarkdown14.1 KB
references/resolve.mdmarkdown8.8 KB
references/setup-providers.mdmarkdown7.1 KB
references/telemetry-dashboard.mdmarkdown4.1 KB
scripts/audio-duck.mjsscript3.9 KB
scripts/dither.mjsscript9.7 KB
scripts/dither.test.mjsscript4.6 KB
scripts/eval.mjsscript14.5 KB
scripts/lib/adopt.mjsscript4.3 KB
scripts/lib/adopt.test.mjsscript2.7 KB
scripts/lib/bgm-provider.mjsscript650 B
scripts/lib/brand-provider.mjsscript1.6 KB
scripts/lib/bundled-sfx-provider.mjsscript3.5 KB
scripts/lib/bundled-sfx-provider.test.mjsscript3.1 KB
scripts/lib/cache.mjsscript4.8 KB
scripts/lib/candidates.mjsscript4.0 KB
scripts/lib/candidates.test.mjsscript5.9 KB
scripts/lib/codex-provider.mjsscript6.5 KB
scripts/lib/codex-provider.test.mjsscript1.3 KB
scripts/lib/coverage.test.mjsscript5.6 KB
scripts/lib/cube-build.mjsscript5.7 KB
scripts/lib/cube-build.test.mjsscript2.7 KB
scripts/lib/cube-validate.mjsscript5.9 KB
scripts/lib/cube-validate.test.mjsscript3.2 KB
scripts/lib/cutlist.mjsscript6.0 KB
scripts/lib/cutlist.test.mjsscript4.0 KB
scripts/lib/duck.mjsscript3.0 KB
scripts/lib/duck.test.mjsscript3.4 KB
scripts/lib/error-diffusion.mjsscript6.5 KB
scripts/lib/error-diffusion.test.mjsscript3.4 KB
scripts/lib/freeze.mjsscript3.0 KB
scripts/lib/freeze.test.mjsscript1.9 KB
scripts/lib/grade-analyzer.mjsscript8.6 KB
scripts/lib/grade-analyzer.test.mjsscript7.7 KB
scripts/lib/heygen-cli.mjsscript7.3 KB
scripts/lib/heygen-cli.test.mjsscript10.0 KB
scripts/lib/heygen-search.mjsscript1.8 KB
scripts/lib/heygen-search.test.mjsscript1.8 KB
scripts/lib/heygen-video-provider.mjsscript3.6 KB
scripts/lib/heygen-video-provider.test.mjsscript12.0 KB
scripts/lib/image-provider.mjsscript1.4 KB
scripts/lib/index-gen.mjsscript1.9 KB
scripts/lib/local-models.mjsscript13.3 KB
scripts/lib/local-models.test.mjsscript5.6 KB
scripts/lib/local-run.mjsscript2.2 KB
scripts/lib/local-run.test.mjsscript1.7 KB
scripts/lib/logo-provider.mjsscript7.0 KB
scripts/lib/logo-provider.test.mjsscript5.7 KB
scripts/lib/ltx-video-provider.mjsscript1.9 KB
scripts/lib/ltx-video-provider.test.mjsscript3.8 KB
scripts/lib/lut-preset-provider.mjsscript8.5 KB
scripts/lib/lut-preset-provider.test.mjsscript6.3 KB
scripts/lib/manifest.mjsscript7.0 KB
scripts/lib/manifest.test.mjsscript11.6 KB
scripts/lib/match.mjsscript1.4 KB
scripts/lib/mflux-provider.mjsscript3.2 KB
scripts/lib/misses.mjsscript1.2 KB
scripts/lib/misses.test.mjsscript2.5 KB
scripts/lib/npx-sync.mjsscript1.3 KB
scripts/lib/npx-sync.test.mjsscript2.0 KB
scripts/lib/parakeet-words.mjsscript1.1 KB
scripts/lib/parakeet-words.test.mjsscript1.4 KB
scripts/lib/prefs-store.mjsscript6.4 KB
scripts/lib/prefs-store.test.mjsscript7.2 KB
scripts/lib/probe.mjsscript1.3 KB
scripts/lib/probe.test.mjsscript1.6 KB
scripts/lib/providers.mjsscript327 B
scripts/lib/recipe-store.mjsscript12.7 KB
scripts/lib/recipe-store.test.mjsscript12.5 KB
scripts/lib/registry.mjsscript9.5 KB
scripts/lib/registry.test.mjsscript9.6 KB
scripts/lib/search.mjsscript906 B
scripts/lib/search.test.mjsscript1.2 KB
scripts/lib/sfx-provider.mjsscript700 B
scripts/lib/specs.mjsscript3.2 KB
scripts/lib/specs.test.mjsscript2.7 KB
scripts/lib/stats.mjsscript3.7 KB
scripts/lib/stats.test.mjsscript5.4 KB
scripts/lib/telemetry.mjsscript8.3 KB
scripts/lib/telemetry.test.mjsscript13.1 KB
scripts/lib/tts-local-provider.mjsscript3.7 KB
scripts/lib/tts-local-provider.test.mjsscript2.6 KB
scripts/lib/usage.mjsscript1.8 KB
scripts/lib/usage.test.mjsscript1.7 KB
scripts/lib/voice-provider.mjsscript1.7 KB
scripts/lib/voice-provider.test.mjsscript1.9 KB
scripts/lib/words.mjsscript764 B
scripts/prefs.mjsscript2.4 KB
scripts/recipe.mjsscript4.0 KB
scripts/resolve.mjsscript43.2 KB
scripts/resolve.test.mjsscript35.8 KB
scripts/transcribe.mjsscript6.5 KB
scripts/transcript-cut.mjsscript7.1 KB

audio/scripts/lib/bgm.mjs

View on GitHub
// bgm.mjs — background music for the media audio engine. Two routes, gated the
// same way as TTS/SFX:
//
//   retrieve (default when HeyGen is configured) — search HeyGen's music library
//        by mood, download the top track. Synchronous. assets/bgm/track.mp3.
//   generate (the alternative; the automatic choice when HeyGen is absent) —
//        Lyria (cloud, $GEMINI_API_KEY/$GOOGLE_API_KEY + google-genai) preferred,
//        else local MusicGen (facebook/musicgen-small via transformers). Spawned
//        DETACHED so the engine can return while audio renders; the caller marks
//        bgm_pending and runs wait-bgm.mjs before assembling. assets/bgm/track.wav.
//
// Missing/failed BGM never blocks a render.

import { spawn, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, openSync, closeSync } from "node:fs";
import { join } from "node:path";
import { downloadTo, searchSounds } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";

const r3 = (x) => Number(x.toFixed(3));
const lyriaKey = () => process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || "";

// Default BGM level. Under narration music is a bed that must stay under the
// voice — 0.12 linear ≈ -18 dB. A silent film (no voice) has no voice to duck
// beneath, so BGM sits forward at 0.9. Callers may override per composition.
export const BGM_BED_VOLUME = 0.12;
export const BGM_SILENT_VOLUME = 0.9;
export const bgmDefaultVolume = (hasVoice) => (hasVoice ? BGM_BED_VOLUME : BGM_SILENT_VOLUME);

const BGM_PY_DEPS = ["transformers", "torch", "soundfile", "numpy"];
const BGM_PY_PROBE =
  "import transformers, soundfile, torch, numpy; from transformers import MusicgenForConditionalGeneration";
const LYRIA_PY_DEPS = ["google-genai", "python-dotenv"];
const LYRIA_PY_PROBE = "import google.genai";

function pyOk(probe) {
  const { cmd, args } = pythonInvocation(["-c", probe]);
  return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}
// `python -m pip`, not a bare `pip` binary: a Homebrew/system Python often
// exposes only `python3`/`pip3` on PATH, so a plain `pip` spawn silently
// no-ops (ENOENT) and the documented "auto-installed on demand" path never
// actually installs. `-m pip` also guarantees the packages land in the SAME
// interpreter pyOk() probes — a bare `pip`/`pip3` could resolve to a
// different Python installation than `python3` if more than one is on PATH.
function pipInstall(deps) {
  const { cmd, args } = pythonInvocation(["-m", "pip", "install", "-q", ...deps]);
  return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}

// ── retrieval (HeyGen music library) ──────────────────────────────────────────
export async function retrieveBgm({ query, headers, hyperframesDir, hasVoice }) {
  const q = query || "calm cinematic underscore";
  const results = await searchSounds(q, "music", headers, { limit: 5 });
  if (!results.length) return null;
  const top = results[0];
  const rel = "assets/bgm/track.mp3";
  await downloadTo(top.audio_url, join(hyperframesDir, rel));
  return {
    path: rel,
    volume: bgmDefaultVolume(hasVoice),
    query: q,
    mode: "retrieve",
    duration_s: typeof top.duration === "number" ? r3(top.duration) : null,
  };
}

// ── mood inference (for the generate path's prompt) ──────────────────────────
// Industry base → archetype shape → emotional-arc tiebreaker. Exported so a
// workflow adapter can build a rich prompt from its own narrative metadata; the
// engine also calls it when generate has only a plain mood query.
export function inferBgmPrompt({ blob = "", archetype = "", arc = "", userPrompt = "" } = {}) {
  if (userPrompt) return userPrompt;
  const b = String(blob).toLowerCase();
  let base;
  let bpm;
  if (/\b(crypto|nft|web3|defi|token|blockchain|exchange|wallet|dao)\b/.test(b)) {
    base = "atmospheric electronic, deep bass, futuristic synths, restrained percussion";
    bpm = 100;
  } else if (/\b(finance|fintech|bank|payment|invest|wealth|insurance|treasury)\b/.test(b)) {
    base = "calm cinematic, soft strings, subtle piano, restrained percussion";
    bpm = 92;
  } else if (/\b(creative|agency|design|studio|art|brand|marketing|content)\b/.test(b)) {
    base = "playful electronic, warm pads, light percussion";
    bpm = 115;
  } else {
    base = "uplifting corporate tech, bright modern piano with synth pads";
    bpm = 108;
  }
  const at = String(archetype).toLowerCase();
  const ar = String(arc).toLowerCase();
  if (/\bpas\b|pain.agitate|pain.+solve/.test(at))
    return `${base}, starts with subtle tension then builds to resolution, BPM ${bpm}, transitions from MINOR to MAJOR`;
  if (/\bbab\b|before.after|future.pac|vision/.test(at))
    return `${base}, cinematic and aspirational, steady build with rising energy, BPM ${bpm}, MAJOR`;
  if (/cascade|feature.benefit/.test(at))
    return `${base}, energetic and driving, consistent momentum, BPM ${Math.min(bpm + 10, 128)}, MAJOR`;
  if (/demo.loop|question.+answer/.test(at))
    return `${base}, clean and focused, minimal arrangement, BPM ${Math.max(bpm - 8, 88)}`;
  if (/frustrat|anxiety|overwhelm|tension/.test(ar) && /relief|excite|triumph/.test(ar))
    return `${base}, builds from understated tension to uplifting resolution, BPM ${bpm}, MINOR to MAJOR`;
  if (/excit|awe|power|triumph/.test(ar))
    return `${base}, energetic and confident, BPM ${bpm}, MAJOR`;
  if (/trust|ease|clarity|reassur/.test(ar))
    return `${base}, warm and reassuring, BPM ${Math.max(bpm - 5, 85)}`;
  return `${base}, BPM ${bpm}, MAJOR`;
}

// ── generation (Lyria → MusicGen, detached) ──────────────────────────────────
// Returns a bgmMeta the caller folds into audio_meta:
//   { path, mode, volume, provider, pid, log, target_duration_s, seed_duration_s,
//     loop_count, pending:true }  on success, or  { disabled:true, reason }.
export function generateBgmDetached({
  prompt,
  durationS,
  hyperframesDir,
  lyriaRecipe,
  seedSeconds = 28,
  hasVoice,
}) {
  const rel = "assets/bgm/track.wav";
  const abs = join(hyperframesDir, rel);
  mkdirSync(join(hyperframesDir, "assets", "bgm"), { recursive: true });
  const log = join(hyperframesDir, "assets", "bgm", `bgm-${Date.now()}.log`);
  const targetS = Math.max(1, durationS);
  const baseMeta = { path: rel, mode: null, volume: bgmDefaultVolume(hasVoice), pending: true };

  const lyriaConfigured = !!lyriaKey() && !!lyriaRecipe && existsSync(lyriaRecipe);

  // Make a backend runnable: prefer Lyria when configured (install google-genai
  // on demand), else ensure local MusicGen deps. Installs are synchronous here —
  // generation itself is detached, so the engine still returns promptly.
  if (lyriaConfigured && !pyOk(LYRIA_PY_PROBE)) pipInstall(LYRIA_PY_DEPS);
  const useLyria = lyriaConfigured && pyOk(LYRIA_PY_PROBE);
  if (!useLyria && !pyOk(BGM_PY_PROBE)) pipInstall(BGM_PY_DEPS);

  const fd = openSync(log, "w");
  if (useLyria) {
    const { cmd, args } = pythonInvocation([
      lyriaRecipe,
      "--output",
      abs,
      "--duration",
      String(targetS),
      "--prompt",
      prompt,
    ]);
    const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
    proc.unref();
    closeSync(fd);
    return {
      ...baseMeta,
      mode: "detached-single",
      provider: "lyria",
      pid: proc.pid,
      log,
      target_duration_s: r3(targetS),
    };
  }

  if (pyOk(BGM_PY_PROBE)) {
    const seedS = Math.min(Math.max(seedSeconds, 10), 30);
    const loops = targetS > seedS ? Math.ceil(targetS / seedS) : 1;
    const script = musicgenScript({ prompt, abs, targetS, seedS });
    const { cmd, args } = pythonInvocation(["-c", script]);
    const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
    proc.unref();
    closeSync(fd);
    return {
      ...baseMeta,
      mode: targetS > seedS ? "detached-seed-loop" : "detached-seed-trim",
      provider: "musicgen",
      pid: proc.pid,
      log,
      target_duration_s: r3(targetS),
      seed_duration_s: seedS,
      loop_count: loops,
    };
  }

  closeSync(fd);
  return {
    disabled: true,
    reason: lyriaConfigured
      ? `Lyria configured but google-genai uninstallable, and local MusicGen unavailable (pip install ${BGM_PY_DEPS.join(" ")})`
      : `no Lyria key/recipe and local MusicGen deps unavailable (pip install ${BGM_PY_DEPS.join(" ")})`,
  };
}

// Inline MusicGen: generate ONE seed clip (≤30s to stay under the decoder's
// positional limit), then trim it down or crossfade-loop it up to the target.
function musicgenScript({ prompt, abs, targetS, seedS }) {
  return `
import math, os, sys, traceback
from pathlib import Path
import numpy as np
import soundfile as sf
from transformers import MusicgenForConditionalGeneration, AutoProcessor

prompt = ${JSON.stringify(prompt)}
out_path = ${JSON.stringify(abs)}
target_s = float(${targetS.toFixed(3)})
seed_s = float(${seedS.toFixed(3)})
token_rate = 50
crossfade_s = 0.3

def apply_fade(arr, sr, fade_in_s=0.08, fade_out_s=0.5):
    n_in = min(int(round(fade_in_s * sr)), arr.shape[0] // 2)
    n_out = min(int(round(fade_out_s * sr)), arr.shape[0] // 2)
    if n_in > 1: arr[:n_in] *= np.linspace(0.0, 1.0, n_in, dtype="float32")
    if n_out > 1: arr[-n_out:] *= np.linspace(1.0, 0.0, n_out, dtype="float32")
    return arr

def loop_crossfade(seed, target_len, xf):
    if seed.shape[0] >= target_len: return seed[:target_len]
    xf = min(xf, seed.shape[0] // 2)
    if xf < 1:
        reps = int(math.ceil(target_len / seed.shape[0]))
        return np.tile(seed, reps)[:target_len]
    t = np.linspace(0.0, 1.0, xf, dtype="float32")
    fade_out = np.cos(t * (math.pi / 2)); fade_in = np.sin(t * (math.pi / 2))
    out = seed.copy()
    while out.shape[0] < target_len:
        tail = out[-xf:] * fade_out; head = seed[:xf] * fade_in
        out = np.concatenate([out[:-xf], tail + head, seed[xf:]])
    return out[:target_len]

try:
    Path(os.path.dirname(out_path)).mkdir(parents=True, exist_ok=True)
    processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
    model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
    model.eval()
    sr = int(model.config.audio_encoder.sampling_rate)
    gen_s = min(seed_s, target_s)
    tokens = max(1, int(math.ceil(gen_s * token_rate)))
    print(f"[musicgen] seed dur={gen_s:.2f}s tokens={tokens}", flush=True)
    inputs = processor(text=[prompt], padding=True, return_tensors="pt")
    audio = model.generate(**inputs, max_new_tokens=tokens)
    seed = audio[0, 0].detach().cpu().numpy().astype("float32")
    peak = float(np.max(np.abs(seed)))
    if peak > 1e-6: seed = seed * (0.89 / peak)
    want = max(1, int(round(target_s * sr)))
    if seed.shape[0] >= want:
        final = seed[:want].copy()
    else:
        final = loop_crossfade(seed, want, int(round(crossfade_s * sr)))
    if final.shape[0] < want: final = np.pad(final, (0, want - final.shape[0]))
    else: final = final[:want]
    final = apply_fade(final, sr)
    peak = float(np.max(np.abs(final)))
    if peak > 1.0: final = final / peak
    sf.write(out_path, final, sr)
    print(f"[musicgen] wrote {out_path} samples={final.shape[0]} sr={sr}", flush=True)
except Exception:
    traceback.print_exc(); sys.exit(1)
`;
}
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall