Skip to content

kali20gakki/msAgent / skills/operator/ops-performance-tuning

ops-performance-tuning

Verified

Compile, profile, diagnose, optimize, and compare Ascend NPU operators across CANN versions and A2/A3/A5 for Ascend C, CATLASS, Triton-Ascend, TileLang-Ascend, PyPTO, and SHMEM/MC2. Use when an operator already has a runnable implementation and the user asks for msOpProf collection, bottleneck analysis, source-level tuning, or a reproducible before/after performance report. Do not use as the primary workflow for operator creation, migration, or unresolved correctness failures.

FileKindSize
SKILL.mdskill md15.4 KB
evals/evals.jsonconfig3.6 KB
references/case-routing.mdmarkdown30.0 KB
references/cases/ascendc/pr_ascendc_allgathermm_scalar_align.mdmarkdown1.7 KB
references/cases/ascendc/pr_ascendc_cann_samples_perf_story.mdmarkdown3.9 KB
references/cases/ascendc/pr_ascendc_cgdr_vf_inverse.mdmarkdown1.6 KB
references/cases/ascendc/pr_ascendc_conv3ddx_tque_tbuf.mdmarkdown1.6 KB
references/cases/ascendc/pr_ascendc_layernorm_areg.mdmarkdown1.6 KB
references/cases/ascendc/pr_ascendc_logit_a5_microapi.mdmarkdown1.9 KB
references/cases/ascendc/pr_ascendc_matmul_case_18x.mdmarkdown1.5 KB
references/cases/ascendc/pr_ascendc_matmul_full_load_wiki.mdmarkdown1.6 KB
references/cases/ascendc/pr_ascendc_matmul_vcv_wiki.mdmarkdown1.8 KB
references/cases/ascendc/pr_ascendc_mc2_fusion.mdmarkdown1.4 KB
references/cases/ascendc/pr_ascendc_mhc_24x.mdmarkdown1.2 KB
references/cases/ascendc/pr_ascendc_msopprof_add_block_imbalance.mdmarkdown1.7 KB
references/cases/ascendc/pr_ascendc_quantize_a2_micro_opt.mdmarkdown2.0 KB
references/cases/ascendc/pr_ascendc_rotate_quant_blaze.mdmarkdown2.8 KB
references/cases/ascendc/pr_ascendc_tiling_l2_official.mdmarkdown1.6 KB
references/cases/catlass/pr_catlass_baseline.mdmarkdown1.3 KB
references/cases/catlass/pr_catlass_fixpipe_dualdst.mdmarkdown1.6 KB
references/cases/catlass/pr_catlass_grouped_matmul_tiling.mdmarkdown1.5 KB
references/cases/catlass/pr_catlass_mstuner_3p8x.mdmarkdown1.9 KB
references/cases/catlass/pr_catlass_swizzle_balance.mdmarkdown1.6 KB
references/cases/pypto/pr_pypto_gqa_decode_vector_merge.mdmarkdown2.4 KB
references/cases/pypto/pr_pypto_l2_hit_layout_s13.mdmarkdown1.9 KB
references/cases/pypto/pr_pypto_per_matmul_tile_shapes.mdmarkdown2.4 KB
references/cases/pypto/pr_pypto_small_shape_matmul.mdmarkdown2.3 KB
references/cases/pypto/pr_pypto_weight_none_cacheable.mdmarkdown2.6 KB
references/cases/shmem/pr_shmem_alltoallv_mr152.mdmarkdown2.4 KB
references/cases/shmem/pr_shmem_reduce_scatter_serial.mdmarkdown2.6 KB
references/cases/tilelang/pr_tilelang_benchmark_vs_ascendc.mdmarkdown1.7 KB
references/cases/tilelang/pr_tilelang_compiler_prs.mdmarkdown2.3 KB
references/cases/tilelang/pr_tilelang_fa_pr698.mdmarkdown1.5 KB
references/cases/tilelang/pr_tilelang_gqa_bwd_pr1494.mdmarkdown1.3 KB
references/cases/tilelang/pr_tilelang_persistent_mqa_decode.mdmarkdown2.2 KB
references/cases/tilelang/pr_tilelang_sfa_pr665.mdmarkdown914 B
references/cases/triton/pr_triton_atomic_simd_24x.mdmarkdown1.8 KB
references/cases/triton/pr_triton_autotune_background.mdmarkdown1.0 KB
references/cases/triton/pr_triton_constexpr_recompile.mdmarkdown1.1 KB
references/cases/triton/pr_triton_lgamma_simt.mdmarkdown1.6 KB
references/compile/compile-ascendc.mdmarkdown9.0 KB
references/compile/compile-catlass.mdmarkdown2.4 KB
references/compile/compile-pypto.mdmarkdown2.0 KB
references/compile/compile-shmem.mdmarkdown2.6 KB
references/compile/compile-tilelang.mdmarkdown2.7 KB
references/compile/compile-triton.mdmarkdown4.4 KB
references/diagnose/diagnose-l2-cache.mdmarkdown2.2 KB
references/diagnose/diagnose-occupancy.mdmarkdown4.0 KB
references/diagnose/diagnose-pipeline.mdmarkdown4.1 KB
references/diagnose/diagnose-roofline.mdmarkdown3.1 KB
references/optimize/optimize-api-usage.mdmarkdown1.7 KB
references/optimize/optimize-ascendc-regbase-vf.mdmarkdown5.9 KB
references/optimize/optimize-ascendc-tiling.mdmarkdown935 B
references/optimize/optimize-catlass.mdmarkdown3.0 KB
references/optimize/optimize-data-copy.mdmarkdown870 B
references/optimize/optimize-index.mdmarkdown3.6 KB
references/optimize/optimize-memory-hierarchy.mdmarkdown599 B
references/optimize/optimize-pipeline.mdmarkdown1.0 KB
references/optimize/optimize-pypto.mdmarkdown1.5 KB
references/optimize/optimize-shmem.mdmarkdown5.1 KB
references/optimize/optimize-tilelang.mdmarkdown1.7 KB
references/optimize/optimize-triton.mdmarkdown3.8 KB
references/profile/benchmark-harness.mdmarkdown2.2 KB
references/profile/profile-msopprof.mdmarkdown7.4 KB
references/troubleshooting.mdmarkdown5.8 KB
scripts/bound_analyzer.pyscript9.8 KB
scripts/delta_report.pyscript5.5 KB
scripts/env_check.shscript5.7 KB
scripts/result_saver.pyscript5.8 KB

references/profile/profile-msopprof.md

View on GitHub
# msOpProf 采集与基础数据分析

> 覆盖 torch_npu.profiler、msprof op Task Duration、SHMEMI_PROF、e2e 与 kernel 双口径;跨版本命令路由见 [算子类型路由](../case-routing.md)。

## 1. 特性介绍

基础数据分析指不依赖重型指标的常规采集与统计:算子总耗时、双路径(自定义 vs 标杆)对比、e2e vs kernel 双口径、phase 级分解。它回答的是调优最基础的问题——"这个算子到底花了多少时间、比标杆快还是慢、时间花在哪个阶段",是所有后续深度分析(occupancy / roofline / l2cache / pipe-timeline)的入口和基线。

## 2. 特性功能

按采集途径分四类输出:

- **torch_npu.profiler → `op_statistic.csv`**:输出 profiler 统计;只在工程正式性能口径本身使用 torch_npu.profiler 时作为基线。
- **msprof op → Task Duration**(§1.2,Triton / TileLang / 单算子):取 `OpBasicInfo.csv` 的 Task Duration 作为 kernel 耗时。
- **SHMEMI_PROF 打点 → Device Frame Table**(§1.3,SHMEM 通信算子):`SHMEMI_PROF_START/END(pf_id)` 宏 kernel 侧打点(最多 64 block × 1024 frame),Host 侧初始化后 `aclshmemx_show_prof` 导出;frame 数据含 phase、cycles、avg_us、max_core_us、% of e2e。
- **e2e vs kernel 双口径指标**(§1.4,SHMEM):`e2e_latency_us`(含 staging copy + barrier + kernel + sync,与 HCCL 公平对比用)与 `kernel_latency_us`(纯 kernel,内部定位用),可选第三指标 `comm_only_us`。

派生指标:`algo_bandwidth = logical_payload_bytes / latency`。`bus_bandwidth` 的换算因通信原语和拓扑而异,必须使用目标通信库文档给出的 bus factor;峰值带宽从目标设备与拓扑资料获取,不在 Skill 中固化。

## 3. 如何使用

先做能力探测:优先尝试 `msprof op --help`,不可用时尝试 `msopprof --help`;再从本机帮助确认应用是位置参数还是 `--application`,以及是否支持 `--warm-up`、`--launch-count`、`--kernel-name` 和指标选项。完整模板见 [算子类型路由 §3](../case-routing.md)。

- 命令行:
  - torch_npu.profiler:沿用工程正式 schedule;前后保持相同 warmup/active,每用例每实现独立采集。输出完整用例表,不只展示提升项。
  - msOpProf 单算子(§1.2):常见命令为 `msprof op --kernel-name="<op>_kernel" --launch-count=20 <application> <args>`;TileLang 侧使用 `--kernel-name="main_kernel"` 或真实生成的 kernel 名。`--warm-up` 仅在帮助中存在时使用,否则在应用内部固定 warmup;禁止用 Python/Torch 计时代替正式 kernel 性能口径。
  - msprof op 上板采集(以 Ascend 950PR 上 add_custom 用例的实际用法为例):

    ```bash
    # 输出目录必须先 chmod 755,否则 msprof 报权限错
    mkdir -p msprof_repro/baseline && chmod 755 msprof_repro msprof_repro/baseline
    # 当前官方常见形式
    msprof op --output=./msprof_repro/baseline \
      --kernel-name=add_custom --launch-count=20 \
      ./execute_add_op

    # 某些 CANN 版本支持以下形式;只在本机 --help 存在时使用
    msprof op --output=./msprof_repro/baseline \
      --application="DEVICE_ID=1 ./execute_add_op"
    ```

  - MC2/多 rank:本机帮助明确支持时使用 `msprof op`;否则回退完整 `msprof`。不同 rank 使用独立输出目录并核对全部 kernel。
  - SHMEMI_PROF(§1.3):头文件 `utils/prof/shmemi_prof.h`;采集规范:warmup 轮跳过、多卡取平均;三阶段采集(A 仅 HCCL baseline → B 仅 SHMEM,间隔 ≥30s 新开 shell → C 离线对比;禁止同 shell 混跑)。
- 使用场景:任何调优的第一步——建立基线、双路径对比、判定收益;SHMEM 场景需要 phase 级分解与 e2e/kernel 双口径时用 SHMEMI_PROF。
- 参考示例:torch_npu.profiler 的 LayerNorm(fp16,大 shape)参考用例,自定义实现相对标杆加速比 1.49~1.89。

### A2/A3/A5 采集能力边界

| 能力 | A2 | A3 | A5 |
|---|---:|---:|---:|
| KernelScale / Occupancy / Source | ✓ | ✓ | ✓ |
| MemoryDetail / TimelineDetail | ✓ | ✓ | — |
| PipeTimeline / PcSampling | — | — | ✓ |

该矩阵来自官方 msOpProf 用户指南;安装版本可能早于文档,最终以本机帮助为准。未指定 `--kernel-name` 时,部分版本只采集第一个被调度的算子,多 kernel 程序必须逐个核对 `OpBasicInfo.csv`。

官方参考:[msOpProf 使用说明](https://github.com/Ascend/msopprof/blob/master/docs/en/user_guide/msopprof_usage.md)、[msOpProf 用户指南](https://github.com/Ascend/msopprof/blob/master/docs/en/user_guide/msopprof_user_guide.md)、[Triton-Ascend Profiling](https://github.com/triton-lang/triton-ascend/blob/main/docs/en/debug_guide/profiling.md)。

## 4. 采集完成检查

- 输出目录中包含目标 kernel 的 `OpBasicInfo.csv`,kernel 数量与运行日志一致。
- 每个指标都能追溯到完整命令和本机帮助;不支持或失败的项写入 `partial`。
- baseline 与 optimized 的设备、频率、shape、dtype、TilingKey、blockDim、launch 和指标组一致。
- 短 kernel 同时保留正式 event 计时;msOpProf 绝对耗时只用于结构与趋势解释。

## 5. 实测坑(A5 / CANN 9.1 验证)

1. **`--kernel-name` 匹配的是 C++ mangled 符号名,不是源码函数名**。Ascend C `__global__` kernel(尤其匿名命名空间内的)实际符号形如 `_ZN12_GLOBAL__N_127gelu_eltwise_regbase_kernelEPhS0_`,用 `--kernel-name=gelu_eltwise_regbase_kernel` 会**静默过滤掉全部 launch**——应用照常跑完、只有一行 WARN、产出为空,极易误判为采集成功。正确做法:**先不加过滤全量采集一次,从 `OpBasicInfo.csv` 抄回真实 kernel 名**再决定是否过滤。
2. **输出目录布局随参数组合变化**:`--kill=on --launch-count=1` 时 CSV 直接在 `OPPROF_*/` 下;多 launch/多 kernel 时在 `OPPROF_*/device<N>/<mangled_name>/<idx>/` 下。脚本化取数必须先 `find` 探测,不要写死路径。
3. **fork 多进程程序(SHMEM 多 rank 样例)采集不完整**:实测只能捕到每个进程**第一个** kernel(后续 kernel 无产出),多 kernel 程序的 kernel 清单核对(§4 第一条)会在此场景失败,应标记 `partial` 并改用 SHMEMI_PROF 或完整 `msprof`。
4. **A5 上部分字段恒为 NA**:vector kernel 的 `aiv_mte3_active_bw`、cube kernel 的 `aiv_*` 整列、`MemoryUB.csv` 对 cube kernel 大量列均为 NA;`--aic-metrics=Occupancy` 可能只产出 `OpBasicInfo.csv` 而无独立 occupancy 数据。诊断前先确认本次产出里哪些列有效,不要用 NA 列参与判读。
5. **`--warm-up` 语义未文档化**:默认 warm-up 次数(实测 5)在 replay 模式下会重放应用前 N 次 launch,短程序反复重启进程会放大进程派生/初始化开销,解释 e2e 数字时注意区分。
6. **可忽略的噪声报错**:采集日志中 `[ERROR] <CheckInputFileValid> ... is not a file`、`Failed to load so libprofapi.so`、`child process exited 1` 与 CSV 正常产出可以并存,以 §4 的产出检查为准,不要仅凭 ERROR 行判失败。
7. **多 pipe 占用互补求和≈1 是串行无重叠的典型指纹**(如 cube 0.44+mte2 0.42+mte1 0.18):判读时把各 pipe ratio 加总,接近 1 且无一条饱和 → 优先怀疑流水未重叠而非单 pipe 瓶颈(详见 [diagnose-pipeline.md 判读规则](../diagnose/diagnose-pipeline.md))。
planning-and-task-breakdownaddyosmani/agent-skillsBreaks work into ordered tasks. Use when you have a spec or clear requirements and need to break work into implementable tasks. Use when a task feels too large to start, when you need to estimate scope, or when parallel work is possible.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
screenpipethedotmack/claude-memSearch screen and audio history recorded by Screenpipe and optionally save selected context into claude-mem. Use when the user asks to recall desktop activity, a meeting, or something seen in another app through Screenpipe.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
contact-widgetnexu-io/open-designSelf-contained floating chat widget with welcome screen, social links, meeting button, and message input. Single HTML file, zero dependencies.Stars: 108kUpdated Claude CodeCodexVerifiedInstall
meeting-notesnexu-io/open-designMeeting notes page — title bar with attendees, agenda checklist, decisions block, action items table with owners + dates, and a "next meeting" footer. Use when the brief mentions "meeting notes", "minutes", "1:1 notes", "all-hands recap", or "会议纪要".Stars: 108kUpdated Claude CodeCodexVerifiedInstall