Skip to content

hashgraph-online/awesome-codex-plugins / plugins/beastlabai/multi-llm-plugin/skills/multi-llm

multi-llm

Flagged

Orchestrate plan reviews, task generation, implementation and code reviews across several LLM CLIs in parallel, or ask every model one free-text question about a plan. Use when the user says: review my plan, multi-llm, run a code review, generate tasks from a plan, implement this plan, ask the models about my plan.

FileKindSize
SKILL.mdskill md4.0 KB
apply_code_fixes_orchestrator.pyscript56.6 KB
apply_commit_fixes_orchestrator.pyscript2.3 KB
apply_suggestions_orchestrator.pyscript4.5 KB
apply_task_suggestions_orchestrator.pyscript19.1 KB
ask_orchestrator.pyscript162.5 KB
assets/icons/Anthropic.svgother584 B
assets/icons/Codex.svgother1.3 KB
assets/icons/Copilot.svgother1.5 KB
assets/icons/Cursor.svgother793 B
assets/icons/GoogleGemini.svgother623 B
assets/icons/Goose.svgother3.2 KB
assets/icons/Kilocode.svgother531 B
assets/icons/MiniMax.pngbinary1.5 KB
assets/icons/Mistral.svgother717 B
assets/icons/MoonshotAI.svgother957 B
assets/icons/Nvidia.svgother2.4 KB
assets/icons/OhMyPi.svgother434 B
assets/icons/OpenAI.svgother1.7 KB
assets/icons/Opencode.svgother325 B
assets/icons/Pi.svgother440 B
assets/icons/Qwen.svgother1.7 KB
assets/icons/Zhipu.pngbinary442 B
assets/icons/xAI.svgother343 B
assets/omp/overlay.ymlconfig2.4 KB
check_workflow_prerequisites.pyscript74.5 KB
code_review_orchestrator.pyscript324.0 KB
display_decisions.pyscript9.8 KB
finalize_tracking.pyscript1.3 KB
full_workflow_orchestrator.pyscript58.8 KB
generate_tasks_orchestrator.pyscript44.2 KB
hosts/capabilities.yamlconfig49.9 KB
hosts/codex/plugin-skill.md.tmplother2.8 KB
hosts/codex/shim.md.tmplother7.3 KB
hosts/opencode/plugin-command.md.tmplother2.2 KB
hosts/opencode/shim.md.tmplother6.8 KB
hosts/render_capabilities.pyscript48.4 KB
hosts/snippets/poll-loop.mdmarkdown4.4 KB
hosts/snippets/uv-preflight.mdmarkdown2.2 KB
hosts/uv-missing.txtmarkdown1.5 KB
implement_orchestrator.pyscript156.0 KB
improve_code_orchestrator.pyscript1.6 KB
improve_commits_orchestrator.pyscript2.5 KB
improve_plan_orchestrator.pyscript1.6 KB
init_config.pyscript57.6 KB
instructions/apply-code-fixes.mdmarkdown52.0 KB
instructions/apply-commit-fixes.mdmarkdown10.3 KB
instructions/apply-suggestions.mdmarkdown48.2 KB
instructions/apply-task-suggestions.mdmarkdown53.9 KB
instructions/ask.mdmarkdown27.6 KB
instructions/full-workflow.mdmarkdown24.6 KB
instructions/generate-tasks.mdmarkdown18.1 KB
instructions/implement.mdmarkdown54.0 KB
instructions/improve-code.mdmarkdown25.0 KB
instructions/improve-commits.mdmarkdown24.8 KB
instructions/improve-plan.mdmarkdown26.2 KB
instructions/init-config.mdmarkdown6.6 KB
instructions/review-code.mdmarkdown43.5 KB
instructions/review-commits.mdmarkdown11.4 KB
instructions/review-plan.mdmarkdown38.4 KB
instructions/review-tasks.mdmarkdown37.8 KB
modes.pyscript58.8 KB
multi_llm.pyscript101.1 KB
prompts/ask.txtmarkdown948 B
prompts/code_review.txtmarkdown4.5 KB
prompts/commit_review.txtmarkdown6.2 KB
prompts/consolidate_suggestions.txtmarkdown3.1 KB
prompts/implementation_task.txtmarkdown1.4 KB
prompts/plan_review.txtmarkdown8.0 KB
prompts/task_review.txtmarkdown8.3 KB
providers.yamlconfig19.6 KB
pyproject.tomlmanifest6.8 KB
references/apply-write-executor.mdmarkdown3.0 KB
references/consolidation.mdmarkdown5.6 KB
references/host-capabilities.mdmarkdown13.5 KB
references/human-decision-batch.mdmarkdown17.2 KB
references/implement-prerequisites.mdmarkdown6.0 KB
references/implement-write-executor.mdmarkdown4.9 KB
references/interrupted-run-recovery.mdmarkdown3.1 KB
references/own-check.mdmarkdown4.3 KB
references/resume-detection.mdmarkdown8.5 KB
references/revalidation.mdmarkdown6.9 KB
references/review-code-apply-fixes.mdmarkdown4.3 KB
references/salvage-handling.mdmarkdown12.5 KB
references/summary-format.mdmarkdown10.0 KB
references/uv-check.mdmarkdown5.7 KB
references/wave-batching.mdmarkdown5.2 KB
review_commits_orchestrator.pyscript2.3 KB
review_plan_orchestrator.pyexecutablescript22.3 KB
review_tasks_orchestrator.pyscript10.7 KB
schemas/cascade_record.schema.jsonconfig6.6 KB
schemas/code_review_issues.schema.jsonconfig2.0 KB
schemas/consolidated.schema.jsonconfig6.8 KB
schemas/consolidation_batch.schema.jsonconfig3.7 KB
schemas/generated_tasks.schema.jsonconfig3.1 KB
schemas/host_capabilities.schema.jsonconfig38.3 KB
schemas/human_decision_verdicts.schema.jsonconfig2.6 KB
schemas/result_line.schema.jsonconfig10.1 KB
schemas/review_suggestions.schema.jsonconfig2.4 KB
schemas/state_file.schema.jsonconfig21.5 KB
schemas/status_file.schema.jsonconfig24.9 KB
schemas/status_json.schema.jsonconfig10.8 KB
schemas/subagent_tasks.schema.jsonconfig20.5 KB
schemas/task_decomposition.schema.jsonconfig1.2 KB
schemas/write_batch_result.schema.jsonconfig8.8 KB
templates/config/providers.override.yamlconfig17.2 KB
templates/consolidated_report_template.htmlother44.1 KB
templates/pr_report_template.htmlother181.2 KB
templates/report_template.htmlother67.5 KB
templates/shared/report_core.cssother4.1 KB
templates/shared/report_core.jsscript25.5 KB
update_plan_tasks.pyscript20.9 KB
utils/__init__.pyscript284 B
utils/advisory.pyscript4.8 KB
utils/apply_orchestrator_base.pyscript220.6 KB
utils/apply_output_helpers.pyscript10.9 KB
utils/apply_path_helpers.pyscript683 B
utils/apply_report.pyscript51.2 KB
utils/apply_selection_helpers.pyscript6.8 KB
utils/apply_suggestion_phase.pyscript11.7 KB
utils/atomic_io.pyscript12.9 KB
utils/attempt_classifier.pyscript31.4 KB
utils/backup.pyscript2.0 KB
utils/batch_done.pyscript79.2 KB
utils/batch_ledger.pyscript27.4 KB
utils/batch_prompt.pyscript29.3 KB
utils/batch_prompt_builder.pyscript12.2 KB
utils/batch_reply.pyscript32.9 KB
utils/cascade_archive.pyscript50.6 KB
utils/cascade_ledger.pyscript39.3 KB
utils/cascade_provenance.pyscript22.0 KB
utils/check_records.pyscript59.3 KB
utils/check_reply.pyscript9.5 KB
utils/check_tree.pyscript1.2 KB
utils/cli_common.pyscript91.2 KB
utils/cli_version_cache.pyscript14.0 KB
utils/clock.pyscript3.5 KB
utils/code_fix_batcher.pyscript30.6 KB
utils/codex_trust.pyscript12.9 KB
utils/commit_apply.pyscript73.3 KB
utils/commit_attribution.pyscript21.2 KB
utils/commit_brief.pyscript32.2 KB
utils/commit_front_door.pyscript81.6 KB
utils/commit_paths.pyscript18.0 KB
utils/commit_range.pyscript61.2 KB
utils/commit_report.pyscript19.9 KB
utils/commit_review.pyscript413 B
utils/commit_spelling.pyscript7.9 KB
utils/commit_write_guard.pyscript9.9 KB
utils/concurrency.pyscript11.6 KB
utils/consolidation.pyscript87.3 KB
utils/detach.pyscript186.8 KB
utils/dispatch_spawn.pyscript56.3 KB
utils/doctor.pyscript76.7 KB
utils/env_vars.pyscript20.5 KB
utils/failure_capture.pyscript10.1 KB
utils/file_discovery.pyscript11.0 KB
utils/filtering.pyscript19.9 KB
utils/git_diff.pyscript23.6 KB
utils/git_utils.pyscript86.6 KB
utils/held_folder.pyscript21.3 KB
utils/host_capabilities.pyscript48.0 KB
utils/host_install.pyscript77.6 KB
utils/host_paths.pyscript12.0 KB
utils/host_routes/__init__.pyscript5.7 KB
utils/host_routes/codex.pyscript18.3 KB
utils/host_routes/npm.pyscript12.2 KB
utils/host_shim.pyscript14.7 KB
utils/html_report_generator.pyscript118.9 KB
utils/human_decisions.pyscript38.3 KB
utils/human_task_plan.pyscript30.2 KB
utils/importance.pyscript6.2 KB
utils/improve_cascade.pyscript63.7 KB
utils/improve_cascade_next.pyscript22.2 KB
utils/improve_steps.pyscript32.4 KB
utils/improve_workflow.pyscript114.4 KB
utils/interactive.pyscript31.6 KB
utils/json_extractor.pyscript61.5 KB
utils/judge_vocab.pyscript5.3 KB
utils/known_checks.pyscript37.5 KB
utils/landlock_dispatch.pyscript103.8 KB
utils/landlock_exec.pyscript21.4 KB
utils/launch_limiter.pyscript13.5 KB
utils/llm_client.pyscript126.9 KB
utils/manual_tasks.pyscript14.9 KB
utils/metrics.pyscript68.1 KB
utils/next_phase.pyscript96.1 KB
utils/next_phase_words.pyscript6.4 KB
utils/output_handler.pyscript23.2 KB
utils/phase_walk.pyscript6.4 KB
utils/plan_updater.pyscript17.9 KB
utils/positionals.pyscript5.8 KB
utils/prior_changes.pyscript28.7 KB
utils/probe_state.pyscript17.9 KB
utils/process_capture.pyscript37.6 KB
utils/process_control.pyscript228.7 KB
utils/process_state.pyscript6.6 KB
utils/progress_tracker.pyscript92.2 KB
utils/prompt_channels.pyscript34.1 KB
utils/prompt_files.pyscript24.5 KB

utils/apply_suggestion_phase.py

View on GitHub
"""The class the two suggestion apply phases share.

``apply-suggestions`` and ``apply-task-suggestions`` both apply validated
review suggestions to one shared artifact (the plan, or the task list), and
read their review the same way: ``grouped.json`` and ``validation.json``
under the review's directory, and the user's edits, skips and overrides in
its ``report.md``. Each orchestrator keeps what differs through the hooks
below: its parser and help text, its target and backup, the path taken when
the review left no validation, its batch prompt and edit rules.
"""

from __future__ import annotations

import os
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple

from . import importance as _importance
from .apply_orchestrator_base import ApplyOrchestratorBase, OrchestratorError
from .apply_output_helpers import detail_pointer_lines
from .apply_path_helpers import load_json_file
from .apply_selection_helpers import merge_edited_descriptions
from .judge_vocab import JUDGE_AUTO_DECIDE, JUDGE_DECIDE
from .report_parser import (
    find_edited_descriptions,
    parse_skipped_group_suggestions,
    parse_skipped_groups,
    parse_skipped_suggestions,
    parse_suggestion_validation_overrides,
    parse_validation_overrides_groups,
)
from .state_manager import CURRENT_FORMAT_VERSION, load_groups_payload, stamp_stable_ids
from .suggestion_batcher import SuggestionBatch
from .validation import load_validation_results


class SuggestionApplyPhase(
    ApplyOrchestratorBase[Dict[str, Any], SuggestionBatch], abstract=True,
):
    """An apply phase over a review's grouped suggestions."""

    #: The ``orchestrator_output.json`` key naming the phase directory. A
    #: contract with the phase's instruction file, so each phase spells it.
    phase_dir_key: str = ""
    #: The text summary's heading.
    summary_title: str = ""

    # ----------------------------------------------------------------
    # Inputs
    # ----------------------------------------------------------------

    def load_data(self) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
        """Load grouped suggestions and validation from the review's directory."""
        review_dir = os.path.join(self.out_dir, self.review_subdir)
        validation_path = Path(review_dir) / "validation.json"

        raw = load_json_file(os.path.join(review_dir, "grouped.json"))
        groups: Optional[List[Dict[str, Any]]] = None
        if raw is not None:
            groups = load_groups_payload(raw)
            stamp_stable_ids(groups)

        if not validation_path.exists():
            self._validation_missing(validation_path, groups)

        # The bytes whose round step 9a checked (`held_verdicts`), not a new read.
        validation = load_validation_results(
            validation_path, raw=self.held_verdicts(validation_path))

        if groups is None:
            raise OrchestratorError(
                f"Grouped suggestions not found in {self.out_dir}. "
                f"Run --{self.review_subdir} first to generate grouped suggestions.",
                exit_code=1,
            )

        return groups, validation

    def _validation_missing(
        self, validation_path: Path, groups: Optional[List[Dict[str, Any]]]
    ) -> None:
        """The review left no ``validation.json``: stop. Never returns."""
        raise OrchestratorError(
            f"Validation results not found at {validation_path}. "
            f"Run --{self.review_subdir} first to generate validation results.",
            exit_code=1,
        )

    def parse_user_edits(self, report_path: str) -> Dict[str, Tuple[str, str]]:
        """Detect and return user-edited descriptions from report.md."""
        return find_edited_descriptions(report_path, self.groups)

    def merge_user_edits(
        self,
        groups: List[Dict[str, Any]],
        edited_descriptions: Dict[str, Tuple[str, str]],
    ) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
        """Delegate to shared merge_edited_descriptions helper."""
        return merge_edited_descriptions(groups, edited_descriptions)

    def parse_skips_from_report(
        self, report_path: str
    ) -> Tuple[Set[str], Set[str], Set[str]]:
        """Parse skipped groups and suggestions from report.md."""
        skipped_group_indices = parse_skipped_groups(report_path)
        skipped_suggestion_ids = parse_skipped_group_suggestions(report_path)
        old_skipped_ids = parse_skipped_suggestions(report_path)
        return skipped_group_indices, skipped_suggestion_ids, old_skipped_ids

    def parse_validation_overrides_from_report(
        self, report_path: str
    ) -> Tuple[Dict[str, str], Dict[str, str]]:
        """Parse validation overrides from report.md."""
        group_overrides = parse_validation_overrides_groups(report_path)
        suggestion_overrides = parse_suggestion_validation_overrides(report_path)
        return group_overrides, suggestion_overrides

    # ----------------------------------------------------------------
    # Items and batches
    # ----------------------------------------------------------------

    def format_item_for_output(self, group: Dict[str, Any], index: int) -> Dict[str, Any]:
        """Format a suggestion group for output to the host."""
        suggestions = group.get("suggestions", [])

        if suggestions:
            primary = suggestions[0]
            title = primary.get("title", group.get("theme", "Unknown"))
            desc = primary.get("desc", "")
            suggestion_type = primary.get("type", "modification")
            reference = primary.get("reference", "")
            importance = _importance.normalize(primary.get("importance"))
        else:
            title = group.get("theme", "Unknown")
            desc = ""
            suggestion_type = "modification"
            reference = ""
            importance = "MEDIUM"

        # Combine descriptions if multiple suggestions in group
        if len(suggestions) > 1:
            all_descs = [s.get("desc", "") for s in suggestions if s.get("desc")]
            desc = "\n\n".join(all_descs)

        formatted = {
            "index": index,
            # Stable group identifier so judging subagents / human-decision
            # routing can reference pre-marked items.
            "group_id": group.get("group_hash", ""),
            "title": title,
            "description": desc,
            "type": suggestion_type,
            "reference": reference,
            **self._reference_fields(reference),
            "importance": importance,
            "theme": group.get("theme", ""),
            "category": group.get("category", ""),
            "validation_status": group.get("validation_status", "valid"),
            "validation_reason": group.get("validation_reason", ""),
            "validation_confidence": group.get("validation_confidence", 0.0),
            "models": group.get("models", []),
            "suggestion_count": len(suggestions),
        }
        # Per-item routing flag: reviewer pre-marked this group "Let the agent
        # decide" in the report. Routed to the per-item judge without prompting.
        if group.get(JUDGE_DECIDE):
            formatted["decision_mode"] = JUDGE_AUTO_DECIDE
        # Under `--scope-policy preserve`: the item's validated scope, and a
        # selected remedy as its instruction. Untouched under `expand`.
        return self.scope_annotate(formatted, group)

    def _reference_fields(self, reference: str) -> Dict[str, Any]:
        """Fields an item carries after ``reference``; none by default."""
        return {}

    def generate_batch_prompts(self, batches: List[SuggestionBatch]) -> List[Dict[str, Any]]:
        """Attach the phase's prompt to each batch.

        Returns a list of dicts (batch.to_dict() + prompt) for output assembly.
        """
        return [
            {**batch.to_dict(), "prompt": self._generate_single_batch_prompt(batch)}
            for batch in batches
        ]

    def _generate_single_batch_prompt(self, batch: SuggestionBatch) -> str:
        """The prompt one batch is dispatched with."""
        raise NotImplementedError(
            f"{type(self).__name__} must implement _generate_single_batch_prompt()"
        )

    # ----------------------------------------------------------------
    # Output
    # ----------------------------------------------------------------

    def build_output_json(
        self,
        batches: List[Any],
        *,
        resume_info: Optional[Dict[str, Any]] = None,
    ) -> Dict[str, Any]:
        """Assemble the final orchestrator_output.json payload."""
        phase_dir = os.path.join(self.out_dir, self.phase_name)
        os.makedirs(phase_dir, exist_ok=True)
        target_fields = self._target_output_fields(phase_dir)

        return {
            "format_version": CURRENT_FORMAT_VERSION,
            "plan_file": self.plan_path,
            **target_fields,
            "prefix": self.prefix,
            "output_dir": self.out_dir,
            self.phase_dir_key: phase_dir,
            "timestamp": datetime.now().isoformat(),
            "batches": batches,
            "to_apply": self.formatted_valid,
            "needs_human_review": self.formatted_human,
            "skipped_items": self.formatted_skipped,
            "user_skipped_items": self.user_skipped_items,
            "skipped_count": len(self.skipped),
            "batching_stats": self.batching_stats,
            "human_review_config": self.build_human_review_config(),
            "resume_info": resume_info,
            "edited_descriptions": self.edit_log,
            "summary": {
                "total_groups": len(self.groups),
                "valid_count": len(self.valid),
                "needs_human_count": len(self.needs_human),
                "skipped_count": len(self.skipped),
                "user_skipped_count": len(self.user_skipped_items),
                "batch_count": len(batches),
                "validation_failed_count": sum(
                    1 for g in self.merged if g.get("validation_status") == "validation_failed"
                ),
                "auto_approved_count": sum(
                    1 for g in self.valid if g.get("auto_approved", False)
                ),
                "edited_description_count": len(self.edit_log),
            },
        }

    def _target_output_fields(self, phase_dir: str) -> Dict[str, Any]:
        """Output fields naming the phase's target, placed after
        ``plan_file``; none by default. Called once the phase directory
        exists and before the rest of the payload is read."""
        return {}

    def print_text_summary(self, batches: List[Any], output_path: str) -> None:
        """Print the human-readable summary (``emit_summary`` owns JSON)."""
        print(f"\n{'=' * 60}")
        print(self.summary_title)
        print(f"{'=' * 60}")
        print(f"Plan: {self.plan_path}")
        for line in self._summary_target_lines():
            print(line)
        print(f"Valid suggestions: {len(self.valid)}")
        print(f"Batches: {len(batches)}")
        print(f"Needs human review: {len(self.needs_human)}")
        print(f"Skipped: {len(self.skipped)}")

        if self.batching_stats.get("batching_enabled"):
            print("\nBatching efficiency:")
            print(f"  Subagent calls saved: {self.batching_stats.get('subagent_calls_saved', 0)}")
            print(f"  Efficiency gain: {self.batching_stats.get('efficiency_gain_percent', 0)}%")

        # The counts only: each batch's items are in the files named here, and
        # `util decisions` shows them to the user (instructions, step 3).
        print()
        for line in detail_pointer_lines(output_path):
            print(line)

    def _summary_target_lines(self) -> List[str]:
        """Summary lines naming the phase's target, after the plan's."""
        return []


__all__ = ["SuggestionApplyPhase"]
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall