kali20gakki/msAgent / skills/quantizer/quant-tuning-evaluate
quant-tuning-evaluate
Verified执行模型测评(统一入口)。LLM/VLM 走主流程 scripts/run_evaluation.py + Evaluation YAML;model_family=dit 走 DiT 扩展节(references/dit/evaluate_workflow.md,vbench.py 批量推理产出视频)。
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.9 KB |
| references/dataset_mapping.md | markdown | 1.3 KB |
| references/dit/evaluate_workflow.md | markdown | 19.7 KB |
| references/error_handling.md | markdown | 2.4 KB |
| scripts/run_evaluation.py | script | 4.0 KB |
references/dataset_mapping.md
View on GitHub# 数据集配置映射
AISBench 注册名速查表。
## 常用数据集
| 数据集 | config_name | 说明 |
|--------|-------------|------|
| gsm8k | `gsm8k_gen_0_shot_cot_str` | 数学推理 |
| aime25 | `aime2025_gen_0_shot_chat_prompt` | 高阶数学 |
| aime24 | `aime2024_gen_0_shot_chat_prompt` | 高阶数学 |
| bfcl-simple | `BFCL_gen_simple` | 工具调用 |
| bfcl-hard | `BFCL_gen_hard` | 工具调用(难) |
| mmlu | `mmlu_gen_0_shot` | 知识问答 |
| mmlu-pro | `mmlu_pro_gen_0_shot` | 知识问答(进阶) |
| humaneval | `humaneval_gen_0_shot` | 代码生成 |
| mbpp | `mbpp_gen_0_shot` | 代码生成 |
| truthfulqa | `truthful_qa_gen` | 事实性 |
| winogrande | `winogrande_gen` | 常识推理 |
| arc-challenge | `arc_challenge_gen_0_shot` | 科学推理 |
## 配置格式
```yaml
datasets:
<dataset_key>: # 自定义 key,与 demand.expectations[].dataset 对应
config_name: <str> # AISBench 注册名(上表)
mode: all # 评测模式
chat_template_kwargs: # 可选
thinking: true # 对于 reasoning 模型
```
## 示例
```yaml
datasets:
gsm8k:
config_name: "gsm8k_gen_0_shot_cot_str"
mode: all
aime25:
config_name: "aime2025_gen_0_shot_chat_prompt"
mode: all
chat_template_kwargs:
thinking: true
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |