Skip to content

kali20gakki/msAgent / skills/profiler/ascend-profiler-collect-adaption

ascend-profiler-collect-adaption

Verified

Adapt torch_npu.profiler interface for AI frameworks and training/inference code on Ascend NPU. Use this skill whenever users need to integrate profiling, add performance data collection, or instrument PyTorch code with torch_npu.profiler on Huawei Ascend NPU platforms. Triggers on: "torch_npu profiler", "NPU profiling", "Ascend profiler", "profiler adaptation", "profile training", "profile inference", "add profiler to framework", "性能采集", "profiling 数据", "昇腾 profiler", and any request to add performance monitoring to Ascend NPU code.

references/training_best_practices.md

View on GitHub
# 训练场景 — torch_npu.profiler 适配最佳实践

> 代表框架:MindSpeed-LLM、MindSpeed-MM

## 采集控制方式:配置文件

涉及文件(以 `MindSpeed-LLM` 为例):

| 文件 | 内容 |
|---|---|
| [`mindspeed_llm/fsdp2/utils/profiler.py`](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/mindspeed_llm/fsdp2/utils/profiler.py) | `ProfilerConfig` 配置类 + `ProfilerManager` 类(FSDP2 路径) |
| [`mindspeed_llm/fsdp2/train/trainer.py`](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/mindspeed_llm/fsdp2/train/trainer.py) | `Trainer` 类,在 `__init__` 中构建 `ProfilerConfig`,在 `train()` 中调用 `start/step/stop` |

MindSpeed-LLM 通过 YAML 配置文件控制 profiler,启动时解析为 `ProfilerConfig` 参数配置类。

YAML 配置文件方式:

```yaml
training:
  profile: true
  profile_step_start: 5
  profile_step_end: 15
  profile_ranks: [0]
  profile_level: level1
  profile_with_cpu: true
  profile_save_path: ./profiler_results
```

`ProfilerConfig` 的具体实现如下:

```python
@dataclass
class ProfilerConfig:
    enabled: bool = False
    profile_step_start: int = 0            # 采集起始 step(包含)
    profile_step_end: int = -1             # 采集结束 step(不包含),-1=直到训练结束
    profile_ranks: List[int] = None        # 采集的卡号,默认 None,__post_init__ 中自动转为 [-1]
    profile_level: str = "level1"          # level_none / level0 / level1 / level2
    profile_export_type: str = "db"      # text / db
    profile_data_simplification: bool = False
    profile_with_cpu: bool = False
    profile_with_stack: bool = False
    profile_with_memory: bool = False
    profile_record_shapes: bool = False
    profile_save_path: str = "./profile"
    current_rank: int = 0

    def __post_init__(self):
        if self.profile_ranks is None:
            self.profile_ranks = [-1]

    def is_profiling_rank(self) -> bool:
        if not self.enabled:
            return False
        if -1 in self.profile_ranks:
            return True
        return self.current_rank in self.profile_ranks
```

`ProfilerManager` 接收 `ProfilerConfig` 配置对象作为构造参数,在内部完成参数解析并透传给 `torch_npu.profiler` 底层接口。该类对外暴露 start()、step()、stop() 三个核心方法,供业务注入点在训练的关键路径上调用,具体实现如下:

```python
class ProfilerManager:
    def __init__(self, config: ProfilerConfig):
        self.config = config
        self.profiler = None
        self._started = False

        if not config.is_profiling_rank():   # ProfilerConfig 上的方法,内含 enabled 检查
            return

        Path(config.profile_save_path).mkdir(parents=True, exist_ok=True)

        # level / export_type 字符串 → torch_npu 枚举(dict 查表 + ValueError 校验)
        profiler_level = _str_to_level(config.profile_level)
        profile_export_type = _str_to_export_type(config.profile_export_type)

        # --- Schedule(根据 step 区间内部推导,用户无需关心)---
        if config.profile_step_end == -1:
            active = 1000000
        else:
            active = config.profile_step_end - config.profile_step_start
            if active <= 0:
                raise ValueError("profile_step_end must be > profile_step_start")
        skip_first = max(0, config.profile_step_start - 1)
        warmup = 0 if config.profile_step_start == 0 else 1

        activities = [npu_profiler.ProfilerActivity.NPU]
        if config.profile_with_cpu:
            activities.append(npu_profiler.ProfilerActivity.CPU)

        self.profiler = npu_profiler.profile(
            activities=activities,
            schedule=npu_profiler.schedule(
                wait=0, warmup=warmup, active=active, repeat=1, skip_first=skip_first
            ),
            on_trace_ready=npu_profiler.tensorboard_trace_handler(config.profile_save_path),
            record_shapes=config.profile_record_shapes,
            profile_memory=config.profile_with_memory,
            with_stack=config.profile_with_stack,
            experimental_config=npu_profiler._ExperimentalConfig(
                aic_metrics=npu_profiler.AiCMetrics.PipeUtilization,
                profiler_level=profiler_level,
                export_type=profile_export_type,
                data_simplification=config.profile_data_simplification,
            ),
        )
        # 添加分布式元数据(rank + world_size)
        self.profiler.add_metadata_json("distributed_args", json.dumps({...}))

    def start(self):
        if self.profiler is not None and not self._started:
            self.profiler.start()
            self._started = True
            logger.info_rank0(f"[RANK {self.config.current_rank}] Profiling started.")

    def step(self):
        if self.profiler is not None:
            self.profiler.step()

    def stop(self):
        if self.profiler is not None:
            self.profiler.stop()
            logger.info_rank0(f"[RANK {self.config.current_rank}] Profiling stopped. "
                              f"Trace saved to {self.config.profile_save_path}")
```

`Trainer` 类初始化的时初始化 `ProfilerManager`类,在训练循环入口处调用 `start()` 方法、在每一步训练结束后调用  `step()` 方法,在训练循环结束处调用 `stop()`方法结束profiling采集。

```python
class Trainer:
    def __init__(self, model, optimizer, lr_scheduler, train_dataloader,
                 args, parallel_args, optimization_args, data_args,
                 ckpt_manager, monitor, tokenizer=None):
        # ... 其他字段 ...
        current_rank = dist.get_rank() if dist.is_initialized() else 0
        prof_config = ProfilerConfig(
            enabled=args.profile,              # 及 profile_step_start/end, ranks,
            profile_level=args.profile_level,  # level, export_type, data_simplification,
            # ...                          # with_cpu/stack/memory, record_shapes,
            profile_save_path=args.profile_save_path,  # save_path, current_rank
            current_rank=current_rank,         # ——共 13 个字段从 TrainingArguments 映射
        )
        self.profiler_manager = ProfilerManager(prof_config)

    def train(self, resume_from_checkpoint=None):
        # ... 前置逻辑 ...
        if self.profiler_manager.profiler is not None:
            self.profiler_manager.start()

        for epoch in range(epochs_trained, int(args.num_train_epochs)):
            for update_step in range(start_update_step, total_updates):
                # ... forward/backward/optimizer ...
                self.global_step += 1

                if self.profiler_manager.profiler is not None:
                    self.profiler_manager.step()

        if self.profiler_manager.profiler is not None:
            self.profiler_manager.stop()
```

调用链:YAML配置文件 → `fsdp2_parse_args` → `TrainingArguments` → `Trainer.__init__` 内部构建 `ProfilerConfig` → `ProfilerManager`→ `Trainer.train()` 调用 `start/step/stop`
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexUnverifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall