kali20gakki/msAgent / skills/quantizer/quant-tuning-evaluate
quant-tuning-evaluate
Verified执行模型测评(统一入口)。LLM/VLM 走主流程 scripts/run_evaluation.py + Evaluation YAML;model_family=dit 走 DiT 扩展节(references/dit/evaluate_workflow.md,vbench.py 批量推理产出视频)。
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.9 KB |
| references/dataset_mapping.md | markdown | 1.3 KB |
| references/dit/evaluate_workflow.md | markdown | 19.7 KB |
| references/error_handling.md | markdown | 2.4 KB |
| scripts/run_evaluation.py | script | 4.0 KB |
scripts/run_evaluation.py
View on GitHub# -------------------------------------------------------------------------
# This file is part of the MindStudio project.
# Copyright (c) 2026 Huawei Technologies Co.,Ltd.
#
# MindStudio is licensed under Mulan PSL v2.
# You can use this software according to the terms and conditions of the Mulan PSL v2.
# You may obtain a copy of Mulan PSL v2 at:
#
# http://license.coscl.org.cn/MulanPSL2
#
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
# See the Mulan PSL v2 for more details.
# -------------------------------------------------------------------------
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Run model evaluation from Evaluation YAML (replaces MCP evaluation_run)."""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional
# ---------------------------------------------------------------------------
# Bootstrap: add shared library to sys.path before any cross-skill imports
# ---------------------------------------------------------------------------
_common_dir = Path(__file__).resolve().parents[2] / "msmodelslim-tools-common" / "scripts"
if str(_common_dir) not in sys.path:
sys.path.insert(0, str(_common_dir))
from script_utils import emit_result, ensure_msmodelslim, parse_int_list
from shared import to_device_type # noqa: E402
def _serialize_evaluate_result(evaluate_result: Any) -> Dict[str, Any]:
"""Return a JSON-compatible representation without losing Decimal precision."""
return evaluate_result.model_dump(mode="json")
def run_evaluation(
quant_model_path: str,
evaluate_id: str,
evaluate_config_path: str,
save_path: str,
device: str = "npu",
device_indices: Optional[List[int]] = None,
) -> Dict[str, Any]:
from msmodelslim.app.auto_tuning.evaluation_service_infra import EvaluateContext
from msmodelslim.infra.service_oriented_evaluate_service import (
ServiceOrientedEvaluateService,
ServiceOrientedEvaluateServiceConfig,
)
from msmodelslim.utils.security import yaml_safe_load
try:
evaluate_config_dict = yaml_safe_load(evaluate_config_path)
evaluate_config = ServiceOrientedEvaluateServiceConfig.model_validate(evaluate_config_dict)
evaluate_service = ServiceOrientedEvaluateService()
evaluate_result = evaluate_service.evaluate(
context=EvaluateContext(
evaluate_id=evaluate_id,
device=to_device_type(device),
device_indices=device_indices,
working_dir=Path(save_path),
),
evaluate_config=evaluate_config,
model_path=Path(quant_model_path),
)
return {
"ok": True,
"evaluate_result": _serialize_evaluate_result(evaluate_result),
}
except Exception as exc:
return {"ok": False, "error": str(exc)}
def main() -> int:
parser = argparse.ArgumentParser(description="Run evaluation from Evaluation YAML.")
parser.add_argument("--quant-model-path", required=True)
parser.add_argument("--evaluate-id", required=True)
parser.add_argument("--evaluate-config-path", required=True)
parser.add_argument("--save-path", required=True)
parser.add_argument("--device", default="npu")
parser.add_argument(
"--device-indices",
default=None,
help="Comma-separated indices or JSON array, e.g. 0,1 or [0,1]",
)
args = parser.parse_args()
ensure_msmodelslim()
result = run_evaluation(
quant_model_path=args.quant_model_path,
evaluate_id=args.evaluate_id,
evaluate_config_path=args.evaluate_config_path,
save_path=args.save_path,
device=args.device,
device_indices=parse_int_list(args.device_indices),
)
return emit_result(result)
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |