Skip to content

Ascend/msagent / skills/benchmark/aisbench-dataset-compression-herding

aisbench-dataset-compression-herding

Verified

当用户直接要求进行数据集压缩测评,或在量化调优过程中需要使用 AISBench 评测工具输出数据集精度等场景时使用。 本 Skill 负责:使用 AISBench 数据集压缩代码、用 RBF Kernel Herding 算法生成压缩后的数据子集(coreset),并按用户目标交付两种产物之一: ① 接入量化调优主流程:产出独立的 coreset 数据集配置(不覆盖原始数据),回传「全集 config_name + coreset config_name」给编排层,供 evaluation.yaml 切换; ② 仅数据集压缩测评(不做量化调优):基于 example 模板生成「全集测试脚本」和「子集测试脚本」两份 shell 脚本。 暂时支持 AIME 2025 和 GPQA,其他数据集需要再结合 aisbench tools目录下 herding_coreset_selector 源码进行分析。

FileKindSize
SKILL.mdskill md18.0 KB
example/run_gpqa_fullset.shscript1.5 KB
example/run_gpqa_subset.shscript1.5 KB

example/run_gpqa_fullset.sh

View on GitHub
#!/usr/bin/env bash
# ============================================================
# 将 GPQA 数据集 path 改为「全集」,并支持运行前后修改/还原
#  - 把 3 个配置文件中的 path 改为全集 gpqa
# 用法: bash run_gpqa_fullset.sh
# ============================================================
set -uo pipefail

# ais_bench 源码根目录(配置文件所在目录)
BASE_DIR="<AIS_BENCH_ROOT>"

# 本次目标数据集绝对路径(全集)
TARGET_PATH="<GPQA_FULL_DATASET_PATH>"

# ais_bench 的 --work-dir
WORK_DIR="<WORK_DIR_FOR_FULL_EVAL>"

# 被修改的配置文件
FILES=(
  "$BASE_DIR/benchmark/configs/datasets/gpqa/gpqa_gen_0_shot_str.py"
  "$BASE_DIR/benchmark/configs/datasets/gpqa/gpqa_gen_0_shot_cot_chat_prompt.py"
  "$BASE_DIR/benchmark/configs/datasets/gpqa/gpqa_ppl_0_shot_str.py"
)

# 1. 备份原始 path 值,并改为全集(只替换引号内路径,保留缩进/逗号/注释)
declare -a ORIG_PATHS=()
for f in "${FILES[@]}"; do
  ORIG_PATHS+=("$(sed -n "s/.*path='\([^']*\)'.*/\1/p" "$f" | head -1)")
  sed -i "s#path='[^']*'#path='$TARGET_PATH'#" "$f"
done
echo "[修改完成] 3 个配置文件的 path 已改为全集:$TARGET_PATH"

# 2. 展示修改结果
echo "----- 修改后 path -----"
for f in "${FILES[@]}"; do
  sed -n 's/.*path=/path=/p' "$f"
done

### 执行aisbench命令
# --num-warmups 0 : 跳过 warmup(预热)阶段,直接进入推理
ais_bench --models vllm_api_general_chat --datasets gpqa_gen --work-dir "$WORK_DIR" --dump-eval-details --num-warmups 0
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall