Skip to content

kali20gakki/msAgent / skills/quantizer/quant-tuning-evaluate

quant-tuning-evaluate

Verified

执行模型测评(统一入口)。LLM/VLM 走主流程 scripts/run_evaluation.py + Evaluation YAML;model_family=dit 走 DiT 扩展节(references/dit/evaluate_workflow.md,vbench.py 批量推理产出视频)。

references/dataset_mapping.md

View on GitHub
# 数据集配置映射

AISBench 注册名速查表。

## 常用数据集

| 数据集 | config_name | 说明 |
|--------|-------------|------|
| gsm8k | `gsm8k_gen_0_shot_cot_str` | 数学推理 |
| aime25 | `aime2025_gen_0_shot_chat_prompt` | 高阶数学 |
| aime24 | `aime2024_gen_0_shot_chat_prompt` | 高阶数学 |
| bfcl-simple | `BFCL_gen_simple` | 工具调用 |
| bfcl-hard | `BFCL_gen_hard` | 工具调用(难) |
| mmlu | `mmlu_gen_0_shot` | 知识问答 |
| mmlu-pro | `mmlu_pro_gen_0_shot` | 知识问答(进阶) |
| humaneval | `humaneval_gen_0_shot` | 代码生成 |
| mbpp | `mbpp_gen_0_shot` | 代码生成 |
| truthfulqa | `truthful_qa_gen` | 事实性 |
| winogrande | `winogrande_gen` | 常识推理 |
| arc-challenge | `arc_challenge_gen_0_shot` | 科学推理 |

## 配置格式

```yaml
datasets:
  <dataset_key>:              # 自定义 key,与 demand.expectations[].dataset 对应
    config_name: <str>        # AISBench 注册名(上表)
    mode: all                 # 评测模式
    chat_template_kwargs:     # 可选
      thinking: true          # 对于 reasoning 模型
```

## 示例

```yaml
datasets:
  gsm8k:
    config_name: "gsm8k_gen_0_shot_cot_str"
    mode: all
  aime25:
    config_name: "aime2025_gen_0_shot_chat_prompt"
    mode: all
    chat_template_kwargs:
      thinking: true
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall