Skip to content

hashgraph-online/awesome-codex-plugins / plugins/beastlabai/multi-llm-plugin/skills/multi-llm

multi-llm

Flagged

Orchestrate plan reviews, task generation, implementation and code reviews across several LLM CLIs in parallel, or ask every model one free-text question about a plan. Use when the user says: review my plan, multi-llm, run a code review, generate tasks from a plan, implement this plan, ask the models about my plan.

FileKindSize
SKILL.mdskill md4.0 KB
apply_code_fixes_orchestrator.pyscript56.6 KB
apply_commit_fixes_orchestrator.pyscript2.3 KB
apply_suggestions_orchestrator.pyscript4.5 KB
apply_task_suggestions_orchestrator.pyscript19.1 KB
ask_orchestrator.pyscript162.5 KB
assets/icons/Anthropic.svgother584 B
assets/icons/Codex.svgother1.3 KB
assets/icons/Copilot.svgother1.5 KB
assets/icons/Cursor.svgother793 B
assets/icons/GoogleGemini.svgother623 B
assets/icons/Goose.svgother3.2 KB
assets/icons/Kilocode.svgother531 B
assets/icons/MiniMax.pngbinary1.5 KB
assets/icons/Mistral.svgother717 B
assets/icons/MoonshotAI.svgother957 B
assets/icons/Nvidia.svgother2.4 KB
assets/icons/OhMyPi.svgother434 B
assets/icons/OpenAI.svgother1.7 KB
assets/icons/Opencode.svgother325 B
assets/icons/Pi.svgother440 B
assets/icons/Qwen.svgother1.7 KB
assets/icons/Zhipu.pngbinary442 B
assets/icons/xAI.svgother343 B
assets/omp/overlay.ymlconfig2.4 KB
check_workflow_prerequisites.pyscript74.5 KB
code_review_orchestrator.pyscript324.0 KB
display_decisions.pyscript9.8 KB
finalize_tracking.pyscript1.3 KB
full_workflow_orchestrator.pyscript58.8 KB
generate_tasks_orchestrator.pyscript44.2 KB
hosts/capabilities.yamlconfig49.9 KB
hosts/codex/plugin-skill.md.tmplother2.8 KB
hosts/codex/shim.md.tmplother7.3 KB
hosts/opencode/plugin-command.md.tmplother2.2 KB
hosts/opencode/shim.md.tmplother6.8 KB
hosts/render_capabilities.pyscript48.4 KB
hosts/snippets/poll-loop.mdmarkdown4.4 KB
hosts/snippets/uv-preflight.mdmarkdown2.2 KB
hosts/uv-missing.txtmarkdown1.5 KB
implement_orchestrator.pyscript156.0 KB
improve_code_orchestrator.pyscript1.6 KB
improve_commits_orchestrator.pyscript2.5 KB
improve_plan_orchestrator.pyscript1.6 KB
init_config.pyscript57.6 KB
instructions/apply-code-fixes.mdmarkdown52.0 KB
instructions/apply-commit-fixes.mdmarkdown10.3 KB
instructions/apply-suggestions.mdmarkdown48.2 KB
instructions/apply-task-suggestions.mdmarkdown53.9 KB
instructions/ask.mdmarkdown27.6 KB
instructions/full-workflow.mdmarkdown24.6 KB
instructions/generate-tasks.mdmarkdown18.1 KB
instructions/implement.mdmarkdown54.0 KB
instructions/improve-code.mdmarkdown25.0 KB
instructions/improve-commits.mdmarkdown24.8 KB
instructions/improve-plan.mdmarkdown26.2 KB
instructions/init-config.mdmarkdown6.6 KB
instructions/review-code.mdmarkdown43.5 KB
instructions/review-commits.mdmarkdown11.4 KB
instructions/review-plan.mdmarkdown38.4 KB
instructions/review-tasks.mdmarkdown37.8 KB
modes.pyscript58.8 KB
multi_llm.pyscript101.1 KB
prompts/ask.txtmarkdown948 B
prompts/code_review.txtmarkdown4.5 KB
prompts/commit_review.txtmarkdown6.2 KB
prompts/consolidate_suggestions.txtmarkdown3.1 KB
prompts/implementation_task.txtmarkdown1.4 KB
prompts/plan_review.txtmarkdown8.0 KB
prompts/task_review.txtmarkdown8.3 KB
providers.yamlconfig19.6 KB
pyproject.tomlmanifest6.8 KB
references/apply-write-executor.mdmarkdown3.0 KB
references/consolidation.mdmarkdown5.6 KB
references/host-capabilities.mdmarkdown13.5 KB
references/human-decision-batch.mdmarkdown17.2 KB
references/implement-prerequisites.mdmarkdown6.0 KB
references/implement-write-executor.mdmarkdown4.9 KB
references/interrupted-run-recovery.mdmarkdown3.1 KB
references/own-check.mdmarkdown4.3 KB
references/resume-detection.mdmarkdown8.5 KB
references/revalidation.mdmarkdown6.9 KB
references/review-code-apply-fixes.mdmarkdown4.3 KB
references/salvage-handling.mdmarkdown12.5 KB
references/summary-format.mdmarkdown10.0 KB
references/uv-check.mdmarkdown5.7 KB
references/wave-batching.mdmarkdown5.2 KB
review_commits_orchestrator.pyscript2.3 KB
review_plan_orchestrator.pyexecutablescript22.3 KB
review_tasks_orchestrator.pyscript10.7 KB
schemas/cascade_record.schema.jsonconfig6.6 KB
schemas/code_review_issues.schema.jsonconfig2.0 KB
schemas/consolidated.schema.jsonconfig6.8 KB
schemas/consolidation_batch.schema.jsonconfig3.7 KB
schemas/generated_tasks.schema.jsonconfig3.1 KB
schemas/host_capabilities.schema.jsonconfig38.3 KB
schemas/human_decision_verdicts.schema.jsonconfig2.6 KB
schemas/result_line.schema.jsonconfig10.1 KB
schemas/review_suggestions.schema.jsonconfig2.4 KB
schemas/state_file.schema.jsonconfig21.5 KB
schemas/status_file.schema.jsonconfig24.9 KB
schemas/status_json.schema.jsonconfig10.8 KB
schemas/subagent_tasks.schema.jsonconfig20.5 KB
schemas/task_decomposition.schema.jsonconfig1.2 KB
schemas/write_batch_result.schema.jsonconfig8.8 KB
templates/config/providers.override.yamlconfig17.2 KB
templates/consolidated_report_template.htmlother44.1 KB
templates/pr_report_template.htmlother181.2 KB
templates/report_template.htmlother67.5 KB
templates/shared/report_core.cssother4.1 KB
templates/shared/report_core.jsscript25.5 KB
update_plan_tasks.pyscript20.9 KB
utils/__init__.pyscript284 B
utils/advisory.pyscript4.8 KB
utils/apply_orchestrator_base.pyscript220.6 KB
utils/apply_output_helpers.pyscript10.9 KB
utils/apply_path_helpers.pyscript683 B
utils/apply_report.pyscript51.2 KB
utils/apply_selection_helpers.pyscript6.8 KB
utils/apply_suggestion_phase.pyscript11.7 KB
utils/atomic_io.pyscript12.9 KB
utils/attempt_classifier.pyscript31.4 KB
utils/backup.pyscript2.0 KB
utils/batch_done.pyscript79.2 KB
utils/batch_ledger.pyscript27.4 KB
utils/batch_prompt.pyscript29.3 KB
utils/batch_prompt_builder.pyscript12.2 KB
utils/batch_reply.pyscript32.9 KB
utils/cascade_archive.pyscript50.6 KB
utils/cascade_ledger.pyscript39.3 KB
utils/cascade_provenance.pyscript22.0 KB
utils/check_records.pyscript59.3 KB
utils/check_reply.pyscript9.5 KB
utils/check_tree.pyscript1.2 KB
utils/cli_common.pyscript91.2 KB
utils/cli_version_cache.pyscript14.0 KB
utils/clock.pyscript3.5 KB
utils/code_fix_batcher.pyscript30.6 KB
utils/codex_trust.pyscript12.9 KB
utils/commit_apply.pyscript73.3 KB
utils/commit_attribution.pyscript21.2 KB
utils/commit_brief.pyscript32.2 KB
utils/commit_front_door.pyscript81.6 KB
utils/commit_paths.pyscript18.0 KB
utils/commit_range.pyscript61.2 KB
utils/commit_report.pyscript19.9 KB
utils/commit_review.pyscript413 B
utils/commit_spelling.pyscript7.9 KB
utils/commit_write_guard.pyscript9.9 KB
utils/concurrency.pyscript11.6 KB
utils/consolidation.pyscript87.3 KB
utils/detach.pyscript186.8 KB
utils/dispatch_spawn.pyscript56.3 KB
utils/doctor.pyscript76.7 KB
utils/env_vars.pyscript20.5 KB
utils/failure_capture.pyscript10.1 KB
utils/file_discovery.pyscript11.0 KB
utils/filtering.pyscript19.9 KB
utils/git_diff.pyscript23.6 KB
utils/git_utils.pyscript86.6 KB
utils/held_folder.pyscript21.3 KB
utils/host_capabilities.pyscript48.0 KB
utils/host_install.pyscript77.6 KB
utils/host_paths.pyscript12.0 KB
utils/host_routes/__init__.pyscript5.7 KB
utils/host_routes/codex.pyscript18.3 KB
utils/host_routes/npm.pyscript12.2 KB
utils/host_shim.pyscript14.7 KB
utils/html_report_generator.pyscript118.9 KB
utils/human_decisions.pyscript38.3 KB
utils/human_task_plan.pyscript30.2 KB
utils/importance.pyscript6.2 KB
utils/improve_cascade.pyscript63.7 KB
utils/improve_cascade_next.pyscript22.2 KB
utils/improve_steps.pyscript32.4 KB
utils/improve_workflow.pyscript114.4 KB
utils/interactive.pyscript31.6 KB
utils/json_extractor.pyscript61.5 KB
utils/judge_vocab.pyscript5.3 KB
utils/known_checks.pyscript37.5 KB
utils/landlock_dispatch.pyscript103.8 KB
utils/landlock_exec.pyscript21.4 KB
utils/launch_limiter.pyscript13.5 KB
utils/llm_client.pyscript126.9 KB
utils/manual_tasks.pyscript14.9 KB
utils/metrics.pyscript68.1 KB
utils/next_phase.pyscript96.1 KB
utils/next_phase_words.pyscript6.4 KB
utils/output_handler.pyscript23.2 KB
utils/phase_walk.pyscript6.4 KB
utils/plan_updater.pyscript17.9 KB
utils/positionals.pyscript5.8 KB
utils/prior_changes.pyscript28.7 KB
utils/probe_state.pyscript17.9 KB
utils/process_capture.pyscript37.6 KB
utils/process_control.pyscript228.7 KB
utils/process_state.pyscript6.6 KB
utils/progress_tracker.pyscript92.2 KB
utils/prompt_channels.pyscript34.1 KB
utils/prompt_files.pyscript24.5 KB

utils/check_reply.py

View on GitHub
#!/usr/bin/env python3
"""``util check-reply``: is the reply a host persisted a well-formed batch result?

A native host (Claude Code, Codex, OpenCode driving subagents itself) runs each
validation batch as a subagent and writes the reply to the task's
``output_path``. Nothing checked that file until reaggregation, and a reply that
did not parse was handled by whatever the host improvised: the 2026-10-01 Claude
Code run lost a closing ``]`` on 3 of 14 Sonnet batches, and the host re-ran one,
hand-edited another and "used the corrected one the subagent included" for a
third (plans/2026-10-01-three-host-racing-run.md, issue 11). A hand-repaired
reply is a verdict nobody produced.

This is the check the host runs right after writing a batch, so a bad reply
costs one re-ask of the same subagent instead of a repair::

    util check-reply --tasks-file <phase>/validation_tasks.json [--task-id ID]

For every task it prints one ``[REPLY_OK]``, ``[REPLY_INVALID]`` or
``[REPLY_MISSING]`` line and exits 1 when any is invalid. What it checks, per task:

* the ``output_path`` file exists, is not empty and parses as JSON. No file
  at all is ``[REPLY_MISSING]``: a reply not written yet, which is what a
  batch whose subagent is still running looks like, and never a bad reply to
  re-ask, so on its own it leaves the exit 0 (fix round 28, live run 12,
  Claude Code racing 01: the host checked every batch while wave 2's
  subagents were still running and got three ``[REPLY_INVALID] … no reply
  file`` lines and exit 1);
* for a ``validation``/``revalidation`` task: the document is ``{"groups": [...]}``
  (a bare list is tolerated, as the merge tolerates it), every entry carries a
  ``status`` that is one of the three verdicts, and the entries cover the task's
  ``group_ids`` exactly — a missing, extra or foreign ``group_hash`` is reported
  by name, because the merge discards a foreign hash and marks a missing one
  ``validation_failed``. An entry's id is its ``group_hash``, else its
  ``group_id``: the merge joins on ``group_hash or group_id``, and a check
  stricter than the merge re-asks a subagent for a reply the merge would have
  used, then calls the batch failed (adversarial review, 2026-10-02).

An entry's ``scope_relation``/``scope_reason``/``scope_reference`` (and a
``preserve`` round's ``selected_remedy``) are never checked here: they are
optional, a missing or malformed classification reads ``unclear`` at the merge
(``scope_policy.normalize_scope_fields``), and a re-ask over one would be a
model call the default policy never makes (plans/scope-policy-and-report-tags.md).

Other kinds are only checked for being JSON: their shape is checked by their own
reaggregation. Advisory about nothing else: it never edits or deletes a file.

A reply holding several JSON objects (an answer, then a correction and a second
answer) is named as such (fix round 26) and still re-asked: it is never cut down to
one of them, here or by the host. Which object the subagent meant cannot be read
from the file, and a draft's hashes that cover the batch exactly can still be
swapped against its own verdicts (live run 11, Claude Code review-plan batch 5).
"""

from __future__ import annotations

if __name__ == "__main__" and not __package__:
    # Run by path: put the skill root on sys.path, found from this file's real
    # location as Python finds sys.path[0], then call the package module's entry
    # point with sys.argv untouched, so the program name stays the one typed
    # (docs/development/dispatcher.md#import-safety).
    import os
    import sys

    _UTILS_DIR = os.path.dirname(os.path.realpath(__file__))
    sys.path[:] = [os.path.dirname(_UTILS_DIR)] + [
        p for p in sys.path if os.path.realpath(p) != _UTILS_DIR]
    from utils.check_reply import main

    sys.exit(main())

import argparse
import json
import os
import sys
from typing import Any, Dict, List, Optional, Sequence

from .schema_validator import validation_reply_problems
from .stream_bootstrap import bootstrap_streams

_GROUP_KINDS = ("validation", "revalidation")


def _objects_in(text: str) -> int:
    """How many whole JSON objects the reply text holds one after another,
    prose between them skipped: the count a "not one object" diagnosis names.

    A diagnosis only, for a reply that already failed to parse: nothing is
    ever kept from it. The first object must open the reply and parse; a
    later ``{`` counts when it starts an object that parses. One that does
    not (a brace in the prose between them, such as "{group_hash}", or a
    broken object) is skipped and the search goes on from where it stopped
    parsing, so a broken object is never counted by the pieces before that
    point (fix round 27: the count stopped at the first such ``{``, and a
    correction whose prose held a brace read as one object). A reply whose
    first object does not parse counts 0.
    """
    decoder = json.JSONDecoder()
    start = len(text) - len(text.lstrip())
    count, index = 0, start
    while True:
        try:
            value, end = decoder.raw_decode(text, index)
        except json.JSONDecodeError as exc:
            if count == 0:
                return count
            # Past the point it stopped parsing, and always forward.
            end = max(index + 1, exc.pos)
        else:
            if not isinstance(value, dict):
                return count
            count += 1
        index = text.find("{", end)
        if index < 0:
            return count


#: What a task with no reply on disk prints (fix round 28): the reply is not
#: there yet, which is no failure and nothing to re-ask.
MISSING_MARKER = "[REPLY_MISSING]"


def _reply_path(task: Dict[str, Any], phase_dir: str) -> str:
    return os.path.join(phase_dir, task["output_name"]) if task.get("output_name") \
        else str(task.get("output_path") or "")


def missing_reply_line(task: Dict[str, Any], phase_dir: str) -> Optional[str]:
    """The ``[REPLY_MISSING]`` line for a task with nothing at its reply path,
    or ``None`` when something is there (or the task names no path, which is
    the task file's problem and stays ``[REPLY_INVALID]``)."""
    path = _reply_path(task, phase_dir)
    if not path or os.path.lexists(path):
        return None
    return (
        f"{MISSING_MARKER} {task.get('task_id')}: no reply at {path} yet, as "
        f"for a batch whose subagent has not returned. Once it returns, write "
        f"its reply there and check it then. This is not a bad reply, so there "
        f"is nothing to re-ask."
    )


def check_task_reply(task: Dict[str, Any], phase_dir: str) -> List[str]:
    """Return the problems with one task's persisted reply; empty means fine."""
    path = _reply_path(task, phase_dir)
    if not path or not os.path.isfile(path):
        return [f"no reply file at {path or '(no output_path)'}"]
    try:
        with open(path, "r", encoding="utf-8") as handle:
            text = handle.read()
    except (OSError, UnicodeDecodeError) as exc:
        return [f"reply file unreadable ({exc})"]
    if not text.strip():
        return ["reply file is empty"]
    try:
        document = json.loads(text)
    except json.JSONDecodeError as exc:
        where = f"{exc.msg} at line {exc.lineno} column {exc.colno}"
        objects = _objects_in(text)
        if objects > 1:
            # Fix round 26 (live run 11): an answer, "Wait, the hashes above
            # are swapped …; corrected reply:", and a second answer.
            return [
                f"reply holds {objects} JSON objects, not one ({where}): "
                f"re-ask for the whole reply, printed once; never keep one "
                f"of them yourself"
            ]
        return [f"reply is not valid JSON ({where})"]
    if task.get("kind") not in _GROUP_KINDS:
        return []
    return validation_reply_problems(document, group_ids=task.get("group_ids"))


def main(argv: Optional[Sequence[str]] = None) -> int:
    bootstrap_streams()
    parser = argparse.ArgumentParser(
        prog="multi_llm.py util check-reply",
        description="Check the replies persisted for a task file's batches.",
    )
    parser.add_argument("--tasks-file", required=True,
                        help="a materialized *_tasks.json (validation_tasks.json, ...)")
    parser.add_argument("--task-id", action="append", default=[],
                        help="check only this task (repeatable); default: every task")
    args = parser.parse_args(argv)

    try:
        # Imported here to keep check_reply light.
        from utils.subagent_tasks import tasks_of
        with open(args.tasks_file, "r", encoding="utf-8") as handle:
            document = json.load(handle)
        tasks = tasks_of(document)
    except (OSError, json.JSONDecodeError) as exc:
        print(f"ERROR: cannot read {args.tasks_file}: {exc}", file=sys.stderr)
        return 2
    phase_dir = os.path.dirname(os.path.abspath(args.tasks_file))
    if args.task_id:
        tasks = [task for task in tasks if task.get("task_id") in args.task_id]
        if not tasks:
            print(f"ERROR: no task matches {', '.join(args.task_id)}", file=sys.stderr)
            return 2

    bad = 0
    for task in tasks:
        missing = missing_reply_line(task, phase_dir)
        if missing:
            print(missing)
            continue
        problems = check_task_reply(task, phase_dir)
        if problems:
            bad += 1
            print(f"[REPLY_INVALID] {task.get('task_id')}: " + "; ".join(problems))
        else:
            print(f"[REPLY_OK] {task.get('task_id')}")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 319kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 319kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 319kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 319kUpdated Claude CodeCodexVerifiedInstall