Skip to content

hashgraph-online/awesome-codex-plugins / plugins/beastlabai/multi-llm-plugin/skills/multi-llm

multi-llm

Flagged

Orchestrate plan reviews, task generation, implementation and code reviews across several LLM CLIs in parallel, or ask every model one free-text question about a plan. Use when the user says: review my plan, multi-llm, run a code review, generate tasks from a plan, implement this plan, ask the models about my plan.

FileKindSize
SKILL.mdskill md4.0 KB
apply_code_fixes_orchestrator.pyscript57.0 KB
apply_commit_fixes_orchestrator.pyscript2.3 KB
apply_suggestions_orchestrator.pyscript4.6 KB
apply_task_suggestions_orchestrator.pyscript19.2 KB
ask_orchestrator.pyscript162.5 KB
assets/icons/Anthropic.svgother584 B
assets/icons/Codex.svgother1.3 KB
assets/icons/Copilot.svgother1.5 KB
assets/icons/Cursor.svgother793 B
assets/icons/GoogleGemini.svgother623 B
assets/icons/Goose.svgother3.2 KB
assets/icons/Kilocode.svgother531 B
assets/icons/MiniMax.pngbinary1.5 KB
assets/icons/Mistral.svgother717 B
assets/icons/MoonshotAI.svgother957 B
assets/icons/Nvidia.svgother2.4 KB
assets/icons/OhMyPi.svgother434 B
assets/icons/OpenAI.svgother1.7 KB
assets/icons/Opencode.svgother325 B
assets/icons/Pi.svgother440 B
assets/icons/Qwen.svgother1.7 KB
assets/icons/Zhipu.pngbinary442 B
assets/icons/xAI.svgother343 B
assets/omp/overlay.ymlconfig2.4 KB
check_workflow_prerequisites.pyscript77.8 KB
code_review_orchestrator.pyscript326.3 KB
display_decisions.pyscript9.8 KB
finalize_tracking.pyscript1.3 KB
full_workflow_orchestrator.pyscript69.7 KB
generate_tasks_orchestrator.pyscript44.2 KB
hosts/capabilities.yamlconfig50.2 KB
hosts/codex/plugin-skill.md.tmplother2.8 KB
hosts/codex/shim.md.tmplother7.3 KB
hosts/opencode/plugin-command.md.tmplother2.2 KB
hosts/opencode/shim.md.tmplother6.8 KB
hosts/render_capabilities.pyscript48.4 KB
hosts/snippets/poll-loop.mdmarkdown4.4 KB
hosts/snippets/uv-preflight.mdmarkdown2.2 KB
hosts/uv-missing.txtmarkdown1.5 KB
implement_orchestrator.pyscript156.4 KB
improve_code_orchestrator.pyscript1.6 KB
improve_commits_orchestrator.pyscript2.5 KB
improve_plan_orchestrator.pyscript1.6 KB
init_config.pyscript58.0 KB
instructions/apply-code-fixes.mdmarkdown56.2 KB
instructions/apply-commit-fixes.mdmarkdown10.3 KB
instructions/apply-suggestions.mdmarkdown48.9 KB
instructions/apply-task-suggestions.mdmarkdown54.5 KB
instructions/ask.mdmarkdown27.6 KB
instructions/full-workflow.mdmarkdown25.3 KB
instructions/generate-tasks.mdmarkdown18.1 KB
instructions/implement.mdmarkdown55.8 KB
instructions/improve-code.mdmarkdown25.7 KB
instructions/improve-commits.mdmarkdown25.5 KB
instructions/improve-plan.mdmarkdown26.7 KB
instructions/init-config.mdmarkdown6.6 KB
instructions/review-code.mdmarkdown44.0 KB
instructions/review-commits.mdmarkdown11.4 KB
instructions/review-plan.mdmarkdown38.8 KB
instructions/review-tasks.mdmarkdown38.2 KB
modes.pyscript58.8 KB
multi_llm.pyscript101.1 KB
prompts/ask.txtmarkdown948 B
prompts/code_review.txtmarkdown4.5 KB
prompts/commit_review.txtmarkdown6.2 KB
prompts/consolidate_suggestions.txtmarkdown3.1 KB
prompts/implementation_task.txtmarkdown1.4 KB
prompts/plan_review.txtmarkdown8.0 KB
prompts/task_review.txtmarkdown8.3 KB
providers.yamlconfig20.7 KB
pyproject.tomlmanifest7.0 KB
references/apply-write-executor.mdmarkdown3.5 KB
references/consolidation.mdmarkdown5.6 KB
references/host-capabilities.mdmarkdown13.5 KB
references/human-decision-batch.mdmarkdown17.2 KB
references/implement-prerequisites.mdmarkdown6.0 KB
references/implement-write-executor.mdmarkdown5.4 KB
references/interrupted-run-recovery.mdmarkdown3.1 KB
references/own-check.mdmarkdown4.3 KB
references/resume-detection.mdmarkdown8.5 KB
references/revalidation.mdmarkdown6.9 KB
references/review-code-apply-fixes.mdmarkdown4.3 KB
references/salvage-handling.mdmarkdown12.5 KB
references/summary-format.mdmarkdown10.0 KB
references/uv-check.mdmarkdown5.7 KB
references/wave-batching.mdmarkdown5.2 KB
review_commits_orchestrator.pyscript2.3 KB
review_plan_orchestrator.pyexecutablescript24.6 KB
review_tasks_orchestrator.pyscript10.7 KB
schemas/cascade_record.schema.jsonconfig6.6 KB
schemas/code_review_issues.schema.jsonconfig2.0 KB
schemas/consolidated.schema.jsonconfig6.8 KB
schemas/consolidation_batch.schema.jsonconfig3.7 KB
schemas/generated_tasks.schema.jsonconfig3.1 KB
schemas/host_capabilities.schema.jsonconfig38.3 KB
schemas/human_decision_verdicts.schema.jsonconfig2.6 KB
schemas/result_line.schema.jsonconfig10.1 KB
schemas/review_suggestions.schema.jsonconfig2.4 KB
schemas/state_file.schema.jsonconfig23.3 KB
schemas/status_file.schema.jsonconfig24.9 KB
schemas/status_json.schema.jsonconfig10.8 KB
schemas/subagent_tasks.schema.jsonconfig20.5 KB
schemas/task_decomposition.schema.jsonconfig1.2 KB
schemas/write_batch_result.schema.jsonconfig8.8 KB
templates/config/providers.override.yamlconfig17.8 KB
templates/consolidated_report_template.htmlother44.1 KB
templates/pr_report_template.htmlother181.2 KB
templates/report_template.htmlother67.5 KB
templates/shared/report_core.cssother4.1 KB
templates/shared/report_core.jsscript25.5 KB
update_plan_tasks.pyscript20.9 KB
utils/__init__.pyscript284 B
utils/advisory.pyscript4.8 KB
utils/apply_orchestrator_base.pyscript229.0 KB
utils/apply_output_helpers.pyscript10.9 KB
utils/apply_path_helpers.pyscript683 B
utils/apply_report.pyscript51.6 KB
utils/apply_selection_helpers.pyscript6.8 KB
utils/apply_suggestion_phase.pyscript11.7 KB
utils/atomic_io.pyscript12.9 KB
utils/attempt_classifier.pyscript33.2 KB
utils/backup.pyscript2.0 KB
utils/batch_done.pyscript81.3 KB
utils/batch_ledger.pyscript27.7 KB
utils/batch_prompt.pyscript30.1 KB
utils/batch_prompt_builder.pyscript13.2 KB
utils/batch_reply.pyscript32.9 KB
utils/batch_suppressions.pyscript15.5 KB
utils/cascade_archive.pyscript50.6 KB
utils/cascade_ledger.pyscript39.3 KB
utils/cascade_provenance.pyscript22.0 KB
utils/check_records.pyscript59.7 KB
utils/check_reply.pyscript9.5 KB
utils/check_tree.pyscript1.2 KB
utils/cli_common.pyscript94.3 KB
utils/cli_version_cache.pyscript14.0 KB
utils/clock.pyscript3.5 KB
utils/code_fix_batcher.pyscript31.4 KB
utils/codex_trust.pyscript12.9 KB
utils/commit_apply.pyscript73.3 KB
utils/commit_attribution.pyscript21.2 KB
utils/commit_brief.pyscript32.2 KB
utils/commit_front_door.pyscript81.9 KB
utils/commit_paths.pyscript18.0 KB
utils/commit_range.pyscript61.2 KB
utils/commit_report.pyscript19.9 KB
utils/commit_review.pyscript413 B
utils/commit_spelling.pyscript7.9 KB
utils/commit_write_guard.pyscript9.9 KB
utils/concurrency.pyscript11.6 KB
utils/consolidation.pyscript89.0 KB
utils/detach.pyscript187.9 KB
utils/dispatch_spawn.pyscript62.4 KB
utils/doctor.pyscript76.7 KB
utils/env_vars.pyscript20.5 KB
utils/failure_capture.pyscript10.1 KB
utils/file_discovery.pyscript11.0 KB
utils/filtering.pyscript19.9 KB
utils/fresh_reset.pyscript3.9 KB
utils/git_diff.pyscript23.6 KB
utils/git_utils.pyscript86.6 KB
utils/held_folder.pyscript21.3 KB
utils/host_capabilities.pyscript48.0 KB
utils/host_install.pyscript77.6 KB
utils/host_paths.pyscript12.0 KB
utils/host_routes/__init__.pyscript5.7 KB
utils/host_routes/codex.pyscript18.3 KB
utils/host_routes/npm.pyscript12.2 KB
utils/host_shim.pyscript14.7 KB
utils/html_report_generator.pyscript118.9 KB
utils/human_decisions.pyscript38.3 KB
utils/human_task_plan.pyscript30.2 KB
utils/importance.pyscript6.2 KB
utils/improve_cascade.pyscript63.7 KB
utils/improve_cascade_next.pyscript22.2 KB
utils/improve_steps.pyscript33.1 KB
utils/improve_workflow.pyscript120.9 KB
utils/interactive.pyscript31.6 KB
utils/json_extractor.pyscript83.1 KB
utils/judge_vocab.pyscript5.3 KB
utils/known_checks.pyscript41.6 KB
utils/landlock_dispatch.pyscript103.8 KB
utils/landlock_exec.pyscript21.4 KB
utils/launch_limiter.pyscript13.5 KB
utils/llm_client.pyscript142.7 KB
utils/manual_tasks.pyscript14.9 KB
utils/metrics.pyscript68.1 KB
utils/next_phase.pyscript96.7 KB
utils/next_phase_words.pyscript6.4 KB
utils/output_handler.pyscript23.2 KB
utils/phase_walk.pyscript6.4 KB
utils/plan_updater.pyscript17.9 KB
utils/positionals.pyscript5.8 KB
utils/prior_changes.pyscript28.7 KB
utils/probe_state.pyscript17.9 KB
utils/process_capture.pyscript44.1 KB
utils/process_control.pyscript228.7 KB
utils/process_state.pyscript6.6 KB
utils/progress_tracker.pyscript92.2 KB

review_tasks_orchestrator.py

View on GitHub
#!/usr/bin/env python3
"""
Orchestrator for reviewing generated implementation tasks against the original plan.

This orchestrator sends the task decomposition to multiple LLMs for quality review,
with plan coverage analysis as the primary concern. It follows the review-plan
pattern as a thin specialization -- all shared review-orchestration logic
(model invocation, result saving, validation, grouping, HTML reports, state
management) lives in utils/review_orchestrator_base.py.

The phase is optional: `--implement` does NOT require `--review-tasks`.

Supports multiple providers (cursor-agent, gemini, opencode) via the provider
registry. Models can be specified as 'provider:model' or bare 'model' names
(which use the default provider from providers.yaml).

Usage:
    # Use YAML defaults (no prompting if defaults.models is set)
    uv run --project "{SKILL_DIR}" -- python "{SKILL_DIR}/multi_llm.py" review-tasks plans/my-plan.md

    # Override defaults with specific models
    uv run --project "{SKILL_DIR}" -- python "{SKILL_DIR}/multi_llm.py" review-tasks plans/my-plan.md --models cursor-agent:auto gemini:gemini-2.5-flash

    # Force interactive selection even if defaults exist
    uv run --project "{SKILL_DIR}" -- python "{SKILL_DIR}/multi_llm.py" review-tasks plans/my-plan.md --interactive
"""

import argparse
import asyncio
import os
import sys
from pathlib import Path
from typing import Dict, List, Optional

# Add parent to path for imports
sys.path.insert(0, str(Path(__file__).parent))

from utils.stream_bootstrap import bootstrap_streams
from utils.cli_common import (
    add_retired_flag,
    build_parent_parser,
    dispatcher_prefix,
    add_scope_policy_argument,
)
from utils.output_handler import output_dir_path
from utils.prompt_loader import load_prompt
from utils.provider_registry import get_all_model_specs
from utils.review_orchestrator_base import (
    derive_prefix,
    run_review_orchestration,
)
from utils.tasks_file import find_tasks_file
from utils import unattended as unattended_mod
from check_workflow_prerequisites import build_unmet_message


# Phase configuration constants
PHASE_NAME = 'review-tasks'
INVOKE_PHASE = 'task_review'
SALVAGE_PHASE_NAME = 'review_tasks'
ORCHESTRATOR_SCRIPT = 'review_tasks_orchestrator.py'
TASK_REVIEW_PROMPT_FILE = "task_review.txt"


def load_task_review_prompt_template() -> str:
    """Load the task review prompt template.

    Returns:
        Prompt template string with placeholders for substitution
    """
    return load_prompt(TASK_REVIEW_PROMPT_FILE)


def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
    """Parse command line arguments for task review."""
    try:
        available_models = get_all_model_specs()
    except FileNotFoundError:
        available_models = ["(providers.yaml not found - run with --models to specify)"]
    except Exception as e:
        available_models = [f"(config error: {type(e).__name__} - check providers.yaml)"]

    # Printed commands, so resolved from `__file__` and never a skill-dir
    # placeholder (see review_plan_orchestrator.parse_args for why).
    run = dispatcher_prefix()
    parser = argparse.ArgumentParser(
        description="Review generated implementation tasks against the original plan",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        parents=[build_parent_parser()],
        epilog=f"""
Available models (provider:model format):
  {', '.join(available_models[:10])}{'...' if len(available_models) > 10 else ''}

Model selection priority:
  1. --models flag       -> Use specified models
  2. --interactive flag  -> Force two-step interactive selection
  3. --quick flag        -> Use quick_models from providers.yaml (2 models)
  4. YAML defaults       -> Use configured defaults from providers.yaml
  5. Interactive         -> Two-step selection (fallback if no defaults)

Examples:
  # Use YAML defaults (no prompting)
  {run} review-tasks plans/my-plan.md

  # Specify models explicitly
  {run} review-tasks plans/my-plan.md --models cursor-agent:auto gemini:gemini-2.5-flash

  # Force interactive selection
  {run} review-tasks plans/my-plan.md --interactive
        """
    )

    parser.add_argument(
        '--plan-file',
        required=True,
        help='Path to the implementation plan markdown file'
    )

    parser.add_argument(
        '--models',
        nargs='+',
        default=None,
        help='List of models in provider:model format (e.g., cursor-agent:auto gemini:gemini-2.5-flash). '
             'Bare model names use default provider from providers.yaml.'
    )

    parser.add_argument(
        '--quick', '-q',
        action='store_true',
        help='Use quick_models from providers.yaml for lightweight reviews (2 models)'
    )

    parser.add_argument(
        '--timeout',
        type=int,
        default=None,
        help='Override timeout per model in seconds (default: use per-provider timeout from providers.yaml)'
    )

    parser.add_argument(
        '--max-parallel',
        type=int,
        default=5,
        help='Maximum number of parallel model invocations (default: 5)'
    )

    parser.add_argument(
        '--skip-validation',
        action='store_true',
        help='Skip the validation step (faster but no false-positive filtering)'
    )

    parser.add_argument(
        '--validation-model',
        type=str,
        default='auto',
        help='Model to use for validation (default: auto)'
    )

    parser.add_argument(
        '--reaggregate',
        action='store_true',
        help='Re-aggregate existing model results (use after salvage operations complete)'
    )

    add_retired_flag(parser, '--internal-validation')

    parser.add_argument(
        '--force',
        action='store_true',
        help='Bypass the completed-phase and partial-completion guards and resume '
             'the phase (already-completed per-model results are kept; only missing '
             'models re-run). For a full re-run that discards existing results, also '
             'pass --rerun-all.'
    )

    parser.add_argument(
        '--rerun-all',
        action='store_true',
        help='Re-run every model from scratch, discarding any existing per-model '
             'result files (default: resume — skip models that already have results).'
    )

    parser.add_argument(
        '--report-style',
        choices=['pr', 'flat'],
        default='pr',
        help='HTML report template style: "pr" for PR-style contextual view, "flat" for classic card layout (default: pr)'
    )

    # plans/scope-policy-and-report-tags.md §4: declared once, in cli_common.
    add_scope_policy_argument(parser, role="review")

    return parser.parse_args(argv)


async def main(argv: Optional[List[str]] = None) -> int:
    """Main entry point."""
    # Line buffering + UTF-8/replace stream encoding (Windows-safe output);
    # see utils/stream_bootstrap.py for the full rationale.
    bootstrap_streams()

    args = parse_args(argv)

    # Load the prompt template
    prompt_template = load_task_review_prompt_template()

    # Resolve plan path early for find_tasks_file
    plan_path_abs = os.path.abspath(args.plan_file)

    # Pre-run existence check: validate tasks file before any model invocation.
    # This is done here rather than in the pre_run_hook because we need the
    # resolved tasks_path for the prompt context factory.
    # In reaggregate mode, we don't need the tasks file (just reprocessing results).
    tasks_path = None
    # Reaggregation re-materializes the same `validation_tasks.json`, whose
    # prompts now *embed* the task list, so this path needs the tasks file after
    # all — but non-fatally, unlike the fan-out: a reaggregate whose tasks file
    # has since moved still has model results worth aggregating, and it ran with
    # no tasks file at all before the text was embedded.
    reaggregate_tasks_path = None
    if not args.reaggregate:
        try:
            tasks_path = find_tasks_file(plan_path_abs)
            print(f"Tasks file: {tasks_path}")
        except (FileNotFoundError, ValueError) as e:
            # D6 row 13: a plan with no task list is "needs a human", not an
            # unexpected error. `check_workflow_prerequisites` owns the gate; this
            # is the orchestrator's own backstop for a dispatcher command invoked
            # directly (which runs no instruction-level prereq check), so it must
            # land on the same exit code and name the same repair — generate the
            # tasks, then re-run — instead of a bare exit 1.
            print(f"ERROR: {e}", file=sys.stderr)
            return unattended_mod.fail_closed(
                build_unmet_message(
                    "review-tasks",
                    Path(plan_path_abs),
                    output_dir_path(Path(plan_path_abs)),
                    [{"phase": "generate-tasks", "reason": str(e)}],
                ),
                site_id="no-tasks-to-review",
                mode="review-tasks",
            )
    else:
        try:
            reaggregate_tasks_path = find_tasks_file(plan_path_abs)
        except (FileNotFoundError, ValueError) as e:
            print(
                f"Warning: {e}; validation will be prepared against the plan alone.",
                file=sys.stderr,
            )

    effective_prefix = derive_prefix(plan_path_abs)

    def prompt_context_factory(model_spec: str) -> Dict:
        return {
            "plan_path": plan_path_abs,
            "tasks_path": tasks_path or "",
            "prefix": effective_prefix,
        }

    # Pass tasks_file as extra validation metadata: it is the artifact under
    # review, so the base embeds its text in every validation prompt (and hashes
    # it into the task file's `inputs`) rather than only recording the path.
    extra_validation_metadata = {}
    validation_tasks_file = tasks_path or reaggregate_tasks_path
    if validation_tasks_file:
        extra_validation_metadata["tasks_file"] = validation_tasks_file

    await run_review_orchestration(
        args=args,
        phase_name=PHASE_NAME,
        invoke_phase=INVOKE_PHASE,
        salvage_phase_name=SALVAGE_PHASE_NAME,
        prompt_template=prompt_template,
        prompt_context_factory=prompt_context_factory,
        orchestrator_script=ORCHESTRATOR_SCRIPT,
        supports_consolidation=False,
        extra_validation_metadata=extra_validation_metadata or None,
        base_ref=None,
        tasks_path=Path(tasks_path) if tasks_path else None,
        template_style=args.report_style,
    )
    # Any non-zero outcome is raised as SystemExit by the orchestration itself;
    # reaching here means the phase completed.
    return 0


if __name__ == "__main__":
    from utils import process_control

    # A process entry of its own: provider children must stay this process's.
    process_control.restore_sigchld_default()
    sys.exit(asyncio.run(main(sys.argv[1:])))
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall