kali20gakki/msAgent / skills/profiler/ascend-profiler-collect-adaption
ascend-profiler-collect-adaption
VerifiedAdapt torch_npu.profiler interface for AI frameworks and training/inference code on Ascend NPU. Use this skill whenever users need to integrate profiling, add performance data collection, or instrument PyTorch code with torch_npu.profiler on Huawei Ascend NPU platforms. Triggers on: "torch_npu profiler", "NPU profiling", "Ascend profiler", "profiler adaptation", "profile training", "profile inference", "add profiler to framework", "性能采集", "profiling 数据", "昇腾 profiler", and any request to add performance monitoring to Ascend NPU code.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.1 KB |
| references/api_reference.md | markdown | 2.1 KB |
| references/inference_best_practices.md | markdown | 5.2 KB |
| references/rl_best_practices.md | markdown | 7.3 KB |
| references/training_best_practices.md | markdown | 7.0 KB |
references/training_best_practices.md
View on GitHub# 训练场景 — torch_npu.profiler 适配最佳实践
> 代表框架:MindSpeed-LLM、MindSpeed-MM
## 采集控制方式:配置文件
涉及文件(以 `MindSpeed-LLM` 为例):
| 文件 | 内容 |
|---|---|
| [`mindspeed_llm/fsdp2/utils/profiler.py`](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/mindspeed_llm/fsdp2/utils/profiler.py) | `ProfilerConfig` 配置类 + `ProfilerManager` 类(FSDP2 路径) |
| [`mindspeed_llm/fsdp2/train/trainer.py`](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/mindspeed_llm/fsdp2/train/trainer.py) | `Trainer` 类,在 `__init__` 中构建 `ProfilerConfig`,在 `train()` 中调用 `start/step/stop` |
MindSpeed-LLM 通过 YAML 配置文件控制 profiler,启动时解析为 `ProfilerConfig` 参数配置类。
YAML 配置文件方式:
```yaml
training:
profile: true
profile_step_start: 5
profile_step_end: 15
profile_ranks: [0]
profile_level: level1
profile_with_cpu: true
profile_save_path: ./profiler_results
```
`ProfilerConfig` 的具体实现如下:
```python
@dataclass
class ProfilerConfig:
enabled: bool = False
profile_step_start: int = 0 # 采集起始 step(包含)
profile_step_end: int = -1 # 采集结束 step(不包含),-1=直到训练结束
profile_ranks: List[int] = None # 采集的卡号,默认 None,__post_init__ 中自动转为 [-1]
profile_level: str = "level1" # level_none / level0 / level1 / level2
profile_export_type: str = "db" # text / db
profile_data_simplification: bool = False
profile_with_cpu: bool = False
profile_with_stack: bool = False
profile_with_memory: bool = False
profile_record_shapes: bool = False
profile_save_path: str = "./profile"
current_rank: int = 0
def __post_init__(self):
if self.profile_ranks is None:
self.profile_ranks = [-1]
def is_profiling_rank(self) -> bool:
if not self.enabled:
return False
if -1 in self.profile_ranks:
return True
return self.current_rank in self.profile_ranks
```
`ProfilerManager` 接收 `ProfilerConfig` 配置对象作为构造参数,在内部完成参数解析并透传给 `torch_npu.profiler` 底层接口。该类对外暴露 start()、step()、stop() 三个核心方法,供业务注入点在训练的关键路径上调用,具体实现如下:
```python
class ProfilerManager:
def __init__(self, config: ProfilerConfig):
self.config = config
self.profiler = None
self._started = False
if not config.is_profiling_rank(): # ProfilerConfig 上的方法,内含 enabled 检查
return
Path(config.profile_save_path).mkdir(parents=True, exist_ok=True)
# level / export_type 字符串 → torch_npu 枚举(dict 查表 + ValueError 校验)
profiler_level = _str_to_level(config.profile_level)
profile_export_type = _str_to_export_type(config.profile_export_type)
# --- Schedule(根据 step 区间内部推导,用户无需关心)---
if config.profile_step_end == -1:
active = 1000000
else:
active = config.profile_step_end - config.profile_step_start
if active <= 0:
raise ValueError("profile_step_end must be > profile_step_start")
skip_first = max(0, config.profile_step_start - 1)
warmup = 0 if config.profile_step_start == 0 else 1
activities = [npu_profiler.ProfilerActivity.NPU]
if config.profile_with_cpu:
activities.append(npu_profiler.ProfilerActivity.CPU)
self.profiler = npu_profiler.profile(
activities=activities,
schedule=npu_profiler.schedule(
wait=0, warmup=warmup, active=active, repeat=1, skip_first=skip_first
),
on_trace_ready=npu_profiler.tensorboard_trace_handler(config.profile_save_path),
record_shapes=config.profile_record_shapes,
profile_memory=config.profile_with_memory,
with_stack=config.profile_with_stack,
experimental_config=npu_profiler._ExperimentalConfig(
aic_metrics=npu_profiler.AiCMetrics.PipeUtilization,
profiler_level=profiler_level,
export_type=profile_export_type,
data_simplification=config.profile_data_simplification,
),
)
# 添加分布式元数据(rank + world_size)
self.profiler.add_metadata_json("distributed_args", json.dumps({...}))
def start(self):
if self.profiler is not None and not self._started:
self.profiler.start()
self._started = True
logger.info_rank0(f"[RANK {self.config.current_rank}] Profiling started.")
def step(self):
if self.profiler is not None:
self.profiler.step()
def stop(self):
if self.profiler is not None:
self.profiler.stop()
logger.info_rank0(f"[RANK {self.config.current_rank}] Profiling stopped. "
f"Trace saved to {self.config.profile_save_path}")
```
`Trainer` 类初始化的时初始化 `ProfilerManager`类,在训练循环入口处调用 `start()` 方法、在每一步训练结束后调用 `step()` 方法,在训练循环结束处调用 `stop()`方法结束profiling采集。
```python
class Trainer:
def __init__(self, model, optimizer, lr_scheduler, train_dataloader,
args, parallel_args, optimization_args, data_args,
ckpt_manager, monitor, tokenizer=None):
# ... 其他字段 ...
current_rank = dist.get_rank() if dist.is_initialized() else 0
prof_config = ProfilerConfig(
enabled=args.profile, # 及 profile_step_start/end, ranks,
profile_level=args.profile_level, # level, export_type, data_simplification,
# ... # with_cpu/stack/memory, record_shapes,
profile_save_path=args.profile_save_path, # save_path, current_rank
current_rank=current_rank, # ——共 13 个字段从 TrainingArguments 映射
)
self.profiler_manager = ProfilerManager(prof_config)
def train(self, resume_from_checkpoint=None):
# ... 前置逻辑 ...
if self.profiler_manager.profiler is not None:
self.profiler_manager.start()
for epoch in range(epochs_trained, int(args.num_train_epochs)):
for update_step in range(start_update_step, total_updates):
# ... forward/backward/optimizer ...
self.global_step += 1
if self.profiler_manager.profiler is not None:
self.profiler_manager.step()
if self.profiler_manager.profiler is not None:
self.profiler_manager.stop()
```
调用链:YAML配置文件 → `fsdp2_parse_args` → `TrainingArguments` → `Trainer.__init__` 内部构建 `ProfilerConfig` → `ProfilerManager`→ `Trainer.train()` 调用 `start/step/stop`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Unverified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |