Skip to content

hashgraph-online/awesome-codex-plugins / plugins/beastlabai/multi-llm-plugin/skills/multi-llm

multi-llm

Flagged

Orchestrate plan reviews, task generation, implementation and code reviews across several LLM CLIs in parallel, or ask every model one free-text question about a plan. Use when the user says: review my plan, multi-llm, run a code review, generate tasks from a plan, implement this plan, ask the models about my plan.

FileKindSize
SKILL.mdskill md4.0 KB
apply_code_fixes_orchestrator.pyscript56.6 KB
apply_commit_fixes_orchestrator.pyscript2.3 KB
apply_suggestions_orchestrator.pyscript4.5 KB
apply_task_suggestions_orchestrator.pyscript19.1 KB
ask_orchestrator.pyscript162.5 KB
assets/icons/Anthropic.svgother584 B
assets/icons/Codex.svgother1.3 KB
assets/icons/Copilot.svgother1.5 KB
assets/icons/Cursor.svgother793 B
assets/icons/GoogleGemini.svgother623 B
assets/icons/Goose.svgother3.2 KB
assets/icons/Kilocode.svgother531 B
assets/icons/MiniMax.pngbinary1.5 KB
assets/icons/Mistral.svgother717 B
assets/icons/MoonshotAI.svgother957 B
assets/icons/Nvidia.svgother2.4 KB
assets/icons/OhMyPi.svgother434 B
assets/icons/OpenAI.svgother1.7 KB
assets/icons/Opencode.svgother325 B
assets/icons/Pi.svgother440 B
assets/icons/Qwen.svgother1.7 KB
assets/icons/Zhipu.pngbinary442 B
assets/icons/xAI.svgother343 B
assets/omp/overlay.ymlconfig2.4 KB
check_workflow_prerequisites.pyscript74.5 KB
code_review_orchestrator.pyscript324.0 KB
display_decisions.pyscript9.8 KB
finalize_tracking.pyscript1.3 KB
full_workflow_orchestrator.pyscript58.8 KB
generate_tasks_orchestrator.pyscript44.2 KB
hosts/capabilities.yamlconfig49.9 KB
hosts/codex/plugin-skill.md.tmplother2.8 KB
hosts/codex/shim.md.tmplother7.3 KB
hosts/opencode/plugin-command.md.tmplother2.2 KB
hosts/opencode/shim.md.tmplother6.8 KB
hosts/render_capabilities.pyscript48.4 KB
hosts/snippets/poll-loop.mdmarkdown4.4 KB
hosts/snippets/uv-preflight.mdmarkdown2.2 KB
hosts/uv-missing.txtmarkdown1.5 KB
implement_orchestrator.pyscript156.0 KB
improve_code_orchestrator.pyscript1.6 KB
improve_commits_orchestrator.pyscript2.5 KB
improve_plan_orchestrator.pyscript1.6 KB
init_config.pyscript57.6 KB
instructions/apply-code-fixes.mdmarkdown52.0 KB
instructions/apply-commit-fixes.mdmarkdown10.3 KB
instructions/apply-suggestions.mdmarkdown48.2 KB
instructions/apply-task-suggestions.mdmarkdown53.9 KB
instructions/ask.mdmarkdown27.6 KB
instructions/full-workflow.mdmarkdown24.6 KB
instructions/generate-tasks.mdmarkdown18.1 KB
instructions/implement.mdmarkdown54.0 KB
instructions/improve-code.mdmarkdown25.0 KB
instructions/improve-commits.mdmarkdown24.8 KB
instructions/improve-plan.mdmarkdown26.2 KB
instructions/init-config.mdmarkdown6.6 KB
instructions/review-code.mdmarkdown43.5 KB
instructions/review-commits.mdmarkdown11.4 KB
instructions/review-plan.mdmarkdown38.4 KB
instructions/review-tasks.mdmarkdown37.8 KB
modes.pyscript58.8 KB
multi_llm.pyscript101.1 KB
prompts/ask.txtmarkdown948 B
prompts/code_review.txtmarkdown4.5 KB
prompts/commit_review.txtmarkdown6.2 KB
prompts/consolidate_suggestions.txtmarkdown3.1 KB
prompts/implementation_task.txtmarkdown1.4 KB
prompts/plan_review.txtmarkdown8.0 KB
prompts/task_review.txtmarkdown8.3 KB
providers.yamlconfig19.6 KB
pyproject.tomlmanifest6.8 KB
references/apply-write-executor.mdmarkdown3.0 KB
references/consolidation.mdmarkdown5.6 KB
references/host-capabilities.mdmarkdown13.5 KB
references/human-decision-batch.mdmarkdown17.2 KB
references/implement-prerequisites.mdmarkdown6.0 KB
references/implement-write-executor.mdmarkdown4.9 KB
references/interrupted-run-recovery.mdmarkdown3.1 KB
references/own-check.mdmarkdown4.3 KB
references/resume-detection.mdmarkdown8.5 KB
references/revalidation.mdmarkdown6.9 KB
references/review-code-apply-fixes.mdmarkdown4.3 KB
references/salvage-handling.mdmarkdown12.5 KB
references/summary-format.mdmarkdown10.0 KB
references/uv-check.mdmarkdown5.7 KB
references/wave-batching.mdmarkdown5.2 KB
review_commits_orchestrator.pyscript2.3 KB
review_plan_orchestrator.pyexecutablescript22.3 KB
review_tasks_orchestrator.pyscript10.7 KB
schemas/cascade_record.schema.jsonconfig6.6 KB
schemas/code_review_issues.schema.jsonconfig2.0 KB
schemas/consolidated.schema.jsonconfig6.8 KB
schemas/consolidation_batch.schema.jsonconfig3.7 KB
schemas/generated_tasks.schema.jsonconfig3.1 KB
schemas/host_capabilities.schema.jsonconfig38.3 KB
schemas/human_decision_verdicts.schema.jsonconfig2.6 KB
schemas/result_line.schema.jsonconfig10.1 KB
schemas/review_suggestions.schema.jsonconfig2.4 KB
schemas/state_file.schema.jsonconfig21.5 KB
schemas/status_file.schema.jsonconfig24.9 KB
schemas/status_json.schema.jsonconfig10.8 KB
schemas/subagent_tasks.schema.jsonconfig20.5 KB
schemas/task_decomposition.schema.jsonconfig1.2 KB
schemas/write_batch_result.schema.jsonconfig8.8 KB
templates/config/providers.override.yamlconfig17.2 KB
templates/consolidated_report_template.htmlother44.1 KB
templates/pr_report_template.htmlother181.2 KB
templates/report_template.htmlother67.5 KB
templates/shared/report_core.cssother4.1 KB
templates/shared/report_core.jsscript25.5 KB
update_plan_tasks.pyscript20.9 KB
utils/__init__.pyscript284 B
utils/advisory.pyscript4.8 KB
utils/apply_orchestrator_base.pyscript220.6 KB
utils/apply_output_helpers.pyscript10.9 KB
utils/apply_path_helpers.pyscript683 B
utils/apply_report.pyscript51.2 KB
utils/apply_selection_helpers.pyscript6.8 KB
utils/apply_suggestion_phase.pyscript11.7 KB
utils/atomic_io.pyscript12.9 KB
utils/attempt_classifier.pyscript31.4 KB
utils/backup.pyscript2.0 KB
utils/batch_done.pyscript79.2 KB
utils/batch_ledger.pyscript27.4 KB
utils/batch_prompt.pyscript29.3 KB
utils/batch_prompt_builder.pyscript12.2 KB
utils/batch_reply.pyscript32.9 KB
utils/cascade_archive.pyscript50.6 KB
utils/cascade_ledger.pyscript39.3 KB
utils/cascade_provenance.pyscript22.0 KB
utils/check_records.pyscript59.3 KB
utils/check_reply.pyscript9.5 KB
utils/check_tree.pyscript1.2 KB
utils/cli_common.pyscript91.2 KB
utils/cli_version_cache.pyscript14.0 KB
utils/clock.pyscript3.5 KB
utils/code_fix_batcher.pyscript30.6 KB
utils/codex_trust.pyscript12.9 KB
utils/commit_apply.pyscript73.3 KB
utils/commit_attribution.pyscript21.2 KB
utils/commit_brief.pyscript32.2 KB
utils/commit_front_door.pyscript81.6 KB
utils/commit_paths.pyscript18.0 KB
utils/commit_range.pyscript61.2 KB
utils/commit_report.pyscript19.9 KB
utils/commit_review.pyscript413 B
utils/commit_spelling.pyscript7.9 KB
utils/commit_write_guard.pyscript9.9 KB
utils/concurrency.pyscript11.6 KB
utils/consolidation.pyscript87.3 KB
utils/detach.pyscript186.8 KB
utils/dispatch_spawn.pyscript56.3 KB
utils/doctor.pyscript76.7 KB
utils/env_vars.pyscript20.5 KB
utils/failure_capture.pyscript10.1 KB
utils/file_discovery.pyscript11.0 KB
utils/filtering.pyscript19.9 KB
utils/git_diff.pyscript23.6 KB
utils/git_utils.pyscript86.6 KB
utils/held_folder.pyscript21.3 KB
utils/host_capabilities.pyscript48.0 KB
utils/host_install.pyscript77.6 KB
utils/host_paths.pyscript12.0 KB
utils/host_routes/__init__.pyscript5.7 KB
utils/host_routes/codex.pyscript18.3 KB
utils/host_routes/npm.pyscript12.2 KB
utils/host_shim.pyscript14.7 KB
utils/html_report_generator.pyscript118.9 KB
utils/human_decisions.pyscript38.3 KB
utils/human_task_plan.pyscript30.2 KB
utils/importance.pyscript6.2 KB
utils/improve_cascade.pyscript63.7 KB
utils/improve_cascade_next.pyscript22.2 KB
utils/improve_steps.pyscript32.4 KB
utils/improve_workflow.pyscript114.4 KB
utils/interactive.pyscript31.6 KB
utils/json_extractor.pyscript61.5 KB
utils/judge_vocab.pyscript5.3 KB
utils/known_checks.pyscript37.5 KB
utils/landlock_dispatch.pyscript103.8 KB
utils/landlock_exec.pyscript21.4 KB
utils/launch_limiter.pyscript13.5 KB
utils/llm_client.pyscript126.9 KB
utils/manual_tasks.pyscript14.9 KB
utils/metrics.pyscript68.1 KB
utils/next_phase.pyscript96.1 KB
utils/next_phase_words.pyscript6.4 KB
utils/output_handler.pyscript23.2 KB
utils/phase_walk.pyscript6.4 KB
utils/plan_updater.pyscript17.9 KB
utils/positionals.pyscript5.8 KB
utils/prior_changes.pyscript28.7 KB
utils/probe_state.pyscript17.9 KB
utils/process_capture.pyscript37.6 KB
utils/process_control.pyscript228.7 KB
utils/process_state.pyscript6.6 KB
utils/progress_tracker.pyscript92.2 KB
utils/prompt_channels.pyscript34.1 KB
utils/prompt_files.pyscript24.5 KB

utils/concurrency.py

View on GitHub
"""The concurrency both executor lanes share: launch spacing, backoff, the task gather.

:func:`plan_launch_limiter` spaces two dispatches of one provider by its
stagger (:func:`get_provider_launch_stagger`, :data:`PROVIDER_STAGGER_DELAY`
overriding every adapter); :func:`get_backoff_delay` is the test switch for a
retry's sleep; :func:`run_tasks_concurrent` is the read/analyze executor's
gather, keyed by task id, with the fan-out's semaphores, spacing and overall
timeout. ``review_orchestrator_base.run_models_concurrent`` (the model
fan-out) and the executor (``task_batch``, ``task_dispatch``) read them here,
below the review stack (docs/development/dispatcher.md#import-layers);
``review_orchestrator_base`` re-exports the functions for reading.
"""

import asyncio
import os
from pathlib import Path
from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union

from .launch_limiter import LaunchLimiter
from .provider_registry import (
    get_provider,
    get_provider_max_concurrent,
    get_provider_timeout,
    parse_model_spec,
)


# Launch spacing. The minimum interval between two dispatches of the *same*
# provider is the adapter's own ``LAUNCH_STAGGER_S`` (``utils/providers/base.py``,
# enforced by ``utils/launch_limiter.py``); this module attribute is only a
# uniform override of every adapter's value, ``None`` in production. Tests set
# it (``0.0`` to switch the spacing off, a positive number to force it on
# providers that declare none).
PROVIDER_STAGGER_DELAY: Optional[float] = None


def get_backoff_delay(base_delay: float) -> float:
    """Get backoff delay, respecting test mode for fast retries.

    When MULTI_LLM_TEST_FAST_BACKOFF=1 is set, returns minimal delay (10ms).
    """
    if os.environ.get("MULTI_LLM_TEST_FAST_BACKOFF") == "1":
        return 0.01
    return base_delay


def get_provider_launch_stagger(provider_name: str) -> float:
    """Seconds between two launches of one provider (0 means none).

    Declared by the adapter (``LLMProvider.LAUNCH_STAGGER_S``), not by
    ``providers.yaml``; :data:`PROVIDER_STAGGER_DELAY` overrides every adapter
    when set. A name with no adapter staggers nothing.
    """
    if PROVIDER_STAGGER_DELAY is not None:
        return float(PROVIDER_STAGGER_DELAY)
    provider = get_provider(provider_name)
    if provider is None:
        return 0.0
    return float(getattr(provider, "LAUNCH_STAGGER_S", 0.0) or 0.0)


def plan_launch_limiter(
    provider_names: Sequence[str],
) -> Tuple[LaunchLimiter, float]:
    """One fan-out's launch limiter, and the margin its overall deadline adds.

    The limiter spaces two dispatches of one provider by that provider's
    stagger (:func:`get_provider_launch_stagger`, through
    :func:`get_backoff_delay`); a provider with none is never throttled. Every
    slot of the fan-out shares it, retries included. The margin is the largest
    provider's slot count times its stagger. See
    docs/development/delegation.md#read-and-analyze-executor.
    """
    counts: Dict[str, int] = {}
    for name in provider_names:
        counts[name] = counts.get(name, 0) + 1
    staggers = {name: get_provider_launch_stagger(name) for name in counts}
    limiter = LaunchLimiter({
        name: get_backoff_delay(stagger)
        for name, stagger in staggers.items() if stagger > 0
    })
    overhead = max(
        (counts[name] * stagger for name, stagger in staggers.items()), default=0.0
    )
    return limiter, overhead


def overall_deadline(
    slot_budgets: Sequence[Tuple[str, float]],
    max_parallel: int,
    provider_limits: Mapping[str, Optional[int]],
    stagger_overhead: float,
) -> float:
    """The overall deadline of a gather whose slots queue behind its gates.

    ``slot_budgets`` is one ``(provider, budget)`` per slot. A slot's budget
    starts once it holds its provider's semaphore (``provider_limits``, ``None``
    for no limit) and then the global ``max_parallel`` one, so it can wait
    behind both. Until a slot of provider ``p`` starts, at every moment either
    the global gate is full or every slot holding ``p``'s gate is running, so it
    starts by ``(W - W_p) / max_parallel + (W_p - its budget) / p's gate``,
    where ``W`` sums every slot's budget and ``W_p`` those of ``p``'s slots
    (the two waits add up; neither covers the other). The deadline is the
    latest such finish, plus one largest budget of headroom (kill grace,
    drain, a launch turn waited for inside the gates) and the launch stagger.
    One slot gives ``2 x budget + stagger``.
    See docs/development/providers.md#the-overall-deadline.
    """
    if not slot_budgets:
        return stagger_overhead
    gate = max(1, int(max_parallel or 1))
    total = sum(budget for _, budget in slot_budgets)
    largest = max(budget for _, budget in slot_budgets)
    per_provider: Dict[str, List[float]] = {}
    for provider, budget in slot_budgets:
        per_provider.setdefault(provider, []).append(budget)
    latest = 0.0
    for provider, budgets in per_provider.items():
        limit = provider_limits.get(provider)
        width = gate if limit is None else max(1, min(int(limit), gate))
        own, last = sum(budgets), max(budgets)
        latest = max(latest, (total - own) / gate + (own - last) / width + last)
    return latest + largest + stagger_overhead


async def run_tasks_concurrent(
    tasks: List[Dict[str, Any]],
    per_task_coro_factory: Callable[[Dict[str, Any], str, int], Any],
    *,
    runner_for_task: Callable[[Dict[str, Any]], str],
    timeout: Optional[float] = None,
    max_parallel: int = 4,
    skip_predicate: Optional[Callable[[Dict[str, Any], str, int], Optional[Any]]] = None,
    provider_limits: Optional[Dict[str, int]] = None,
    anchor: Optional[Union[str, Path]] = None,
) -> Dict[str, Any]:
    """``review_orchestrator_base.run_models_concurrent``'s mechanics, keyed by ``task_id`` (D3).

    ``run_models_concurrent`` returns a dict keyed by ``model_spec`` — correct for
    model fan-out, **wrong** for a subagent batch, where every task normally
    shares the one configured runner, so completed tasks would overwrite one
    another. This is the same machinery keyed by each task's stable ``task_id``:

    - the same global ``asyncio.Semaphore(max_parallel)``
    - the same per-provider semaphores from ``get_provider_max_concurrent``,
      acquired *before* the global one, and **shared in shape** with the fan-out
      path so a mixed-tier batch on one provider contends on ONE semaphore
      (the rate limit belongs to the provider, not the model)
    - the same launch spacing, per resolved provider
    - the same overall timeout wrapping ``asyncio.gather``

    ``model_spec`` is used *only* to pick the provider semaphore and launch
    spacing; it is a column on each row, never the key. ``run_models_concurrent``
    keeps its current signature and callers.

    Args:
        tasks: The materialized task objects (not model specs — each task carries
            its own prompt and its runner is a *derived* attribute).
        per_task_coro_factory: ``(task, runner_spec, index) -> awaitable``.
        runner_for_task: Resolves a task's runner spec once, at batch start.
        timeout: Per-task timeout override, also driving the overall timeout.
        max_parallel: The global gate (``driver.max_concurrent_subagents``,
            already reduced to D3i's three-way minimum by the caller).
        skip_predicate: Cheap synchronous ``(task, runner, index) -> result|None``
            checked *before* any semaphore acquisition and launch turn, so a
            resume of already-complete tasks stays fast.
        provider_limits: Optional per-provider in-flight overrides (D3i
            backpressure narrows a throttled provider's slots for the rest of the
            batch); defaults to ``get_provider_max_concurrent``.
        anchor: Optional discovery anchor (the run's plan-file path), the Lane B
            twin of ``run_models_concurrent``'s. ``subagent_executor``
            already holds the phase's plan path and passes it.

    Returns:
        ``{task_id: whatever the callback returned}``.
    """
    results: Dict[str, Any] = {}
    if not tasks:
        return results

    runners = {
        str(task.get("task_id", index)): runner_for_task(task)
        for index, task in enumerate(tasks)
    }

    semaphore = asyncio.Semaphore(max_parallel)

    provider_semaphores: Dict[str, asyncio.Semaphore] = {}
    provider_gates: Dict[str, Optional[int]] = {}
    for spec in set(runners.values()):
        prov, _ = parse_model_spec(spec, anchor=anchor)
        if prov in provider_gates:
            continue
        limit = (provider_limits or {}).get(
            prov, get_provider_max_concurrent(prov, anchor=anchor)
        )
        provider_gates[prov] = limit
        if limit is not None:
            provider_semaphores[prov] = asyncio.Semaphore(limit)

    launcher, stagger_overhead = plan_launch_limiter(
        [
            parse_model_spec(runners[str(task.get("task_id", index))], anchor=anchor)[0]
            for index, task in enumerate(tasks)
        ]
    )

    async def run_with_semaphore(task: Dict[str, Any], index: int) -> Tuple[str, Any]:
        task_id = str(task.get("task_id", index))
        runner = runners[task_id]
        # Anchored for the same reason the fan-out's display resolution is: the
        # semaphore this picks must be the one the batch was sized for.
        provider_name, _ = parse_model_spec(runner, anchor=anchor)

        if skip_predicate is not None:
            skipped = skip_predicate(task, runner, index)
            if skipped is not None:
                return (task_id, skipped)

        # The launch turn is taken inside both semaphores, as on the fan-out;
        # `dispatch_task` confirms or renews it before every attempt.
        prov_sem = provider_semaphores.get(provider_name)
        turn = launcher.slot(provider_name, label=f"{task_id} {runner}")
        if prov_sem:
            async with prov_sem:
                async with semaphore, turn:
                    return (task_id, await per_task_coro_factory(task, runner, index))
        else:
            async with semaphore, turn:
                return (task_id, await per_task_coro_factory(task, runner, index))

    coros = [
        asyncio.create_task(run_with_semaphore(task, idx))
        for idx, task in enumerate(tasks)
    ]

    # `timeout` is the caller's whole-batch budget (`task_batch` sizes it as one
    # task budget per task, since backpressure can narrow a provider to one
    # slot mid-batch), so it counts as one slot; without one, the waits behind
    # the gates the batch starts with are added up.
    if timeout is not None:
        total_timeout = overall_deadline(
            [("", float(timeout))], 1, {}, stagger_overhead
        )
    else:
        total_timeout = overall_deadline(
            [
                (provider, float(get_provider_timeout(provider, anchor=anchor)))
                for provider in (
                    parse_model_spec(spec, anchor=anchor)[0] for spec in runners.values()
                )
            ],
            max_parallel,
            provider_gates,
            stagger_overhead,
        )

    try:
        completed = await asyncio.wait_for(
            asyncio.gather(*coros, return_exceptions=True), timeout=total_timeout
        )
        for item in completed:
            if isinstance(item, Exception):
                print(f"Subagent task exception: {item}")
                continue
            task_id, result = item
            results[task_id] = result
    except asyncio.TimeoutError:
        print(f"Total subagent batch timeout ({total_timeout}s) exceeded")
        for coro in coros:
            if not coro.done():
                coro.cancel()

    return results
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall