kali20gakki/msAgent / skills/quantizer/msmodelslim-model-adapt
msmodelslim-model-adapt
Verified为 msModelSlim 创建基础 Transformers 模型适配器(Model Adapter)。 包含创建适配器、实现必需接口与注册安装流程。 适用:Decoder-only LLM、理解类 VLM(仅 LLM/text 部分)。 多模态生成(DiT/扩散,含双/多专家)见下方「多模态生成扩展」节,按需加载。 不适用:Encoder-only、非 Transformers 架构。
assets/dit/skeleton.md
View on GitHub# 多模态生成(DiT/扩散)双/多专家适配器骨架
> 单 DiT 模板见 `assets/dit_model_adapter_template.py`。
> 双/多专家场景(如 Wan2.2-T2V/I2V/TI2V 系列)按本骨架走「基类 + 场景子类 + ExpertSubAdapter」三层结构。
>
> 三层结构防御通用陷阱(meta tensor、enable_dump 短路、ExitStack、expert key 校验等)。
> **新模型只改 InferenceConfig(分区 2)+ CLI 桥接(分区 6)+ pipeline 加载(分区 7)即可。**
---
## 文件 1:双/多专家基类骨架
`dual_expert_base_model_adapter.py`(参考 Wan2.2 base_model_adapter)
> 在单 DiT 骨架(见 `assets/dit_model_adapter_template.py`)的基础上额外实现:
> - `scene_task: ClassVar[str]` 子类固定;
> - `_bind_expert_sub_adapters(experts)` / `_create_expert_sub_adapter(name)`;
> - `_quantization_context_with_no_sync(*dit_models)` 用 ExitStack;
> - `get_expert_adapter(name)` 在双专家未绑定时 `InvalidModelError`;
> - `_runtime_value` / `_namespace_to_argv` 与单 DiT 共享。
```python
"""双/多专家 DiT 基类骨架(参考 Wan2.2 base_model_adapter)。
子适配器需求(见文件 2):每个 expert 必须有独立的 Wan2_2ExpertSubAdapter。
本类负责:bind 子适配器、按 expert_name 调度、ExitStack 量化上下文、release 辅助模型。
"""
from contextlib import ExitStack, contextmanager, nullcontext
from typing import ClassVar, Dict, Optional, Union
import torch
from pydantic import BaseModel
from torch import nn
from .expert_sub_adapter import MyExpertSubAdapter # 见文件 2
class MyDualExpertBaseAdapter(...):
"""通用多专家 DiT 基类。
子类只负责:
1. 固定 scene_task;
2. 声明 *InferenceConfig(含 task 校验);
3. init_model 中按 DUAL_EXPERT_SCENE_TASKS 写出 experts dict 并调 _bind_expert_sub_adapters;
4. _generate_video 调目标 pipeline;
5. _load_pipeline 创建目标 pipeline 写出 self.low_noise_model / self.high_noise_model。
"""
scene_task: ClassVar[str] = "" # 子类固定
_GENERATE_CONFIG_KEYS: ClassVar[Optional[frozenset[str]]] = None
DUAL_EXPERT_SCENE_TASKS: ClassVar[frozenset[str]] = frozenset() # 子类固定双/多专家场景名
def __init__(self, model_type, model_path, trust_remote_code=False):
if not self.scene_task:
raise SchemaValidateError("Must subclass with scene_task fixed")
super().__init__(model_type, model_path, trust_remote_code)
self.low_noise_model = None
self.high_noise_model = None
self._expert_adapters: Dict[str, MyExpertSubAdapter] = {}
def init_model(self, device=DeviceType.NPU) -> Dict[str, nn.Module]:
"""P6 通用:基类提供,子类覆写后末尾必须调 _bind_expert_sub_adapters。"""
raise NotImplementedError(f"{type(self).__name__} must implement init_model()")
def get_expert_adapter(self, expert_name: str):
"""P6 通用:双专家未绑定时报 InvalidModelError(不静默回退父适配器)。"""
a = self._expert_adapters.get(expert_name)
if a is not None: return a
if self.scene_task not in self.DUAL_EXPERT_SCENE_TASKS and expert_name == "":
return self
raise InvalidModelError(
f"Expert sub-adapter not found for {expert_name!r} (scene_task={self.scene_task!r}).",
action="Ensure init_model() calls _bind_expert_sub_adapters with keys matching QuantService expert names.",
)
# ===== P8 通用:ExitStack 多 expert 量化上下文 =====
@contextmanager
def _quantization_context_with_no_sync(self, *dit_models: nn.Module):
import torch.cuda.amp as amp
for m in dit_models:
if m is None: continue
for name, module in m.named_modules():
if not name: continue
(module.to('cpu') if name.startswith('blocks') else module.to('npu'))
with amp.autocast(dtype=self.model_args.param_dtype), torch.no_grad(), ExitStack() as stack:
for m in dit_models:
if m is None: continue
stack.enter_context(getattr(m, "no_sync", nullcontext)())
yield
# ===== 分区 5:私有专家子适配器装配 =====
def _bind_expert_sub_adapters(self, expert_modules: Dict[str, nn.Module]):
adapters = {}
for name, module in expert_modules.items():
sub = self._create_expert_sub_adapter(name)
sub.bind_module(module)
adapters[name] = sub
self._expert_adapters = adapters
def _create_expert_sub_adapter(self, expert_name: str) -> "MyExpertSubAdapter":
return MyExpertSubAdapter(self, expert_name)
```
---
## 文件 2:双/多专家子适配器骨架
`expert_sub_adapter.py`
```python
"""子适配器骨架(按 expert 调度的代理)。
P7 通用:必须显式继承 OnlineQuaRotInterface / FA3QuantAdapterInterface / IterSmoothInterface,
不能仅靠 __getattr__ 委托 —— LayerWiseRunner 的 isinstance 会失败(详见 pitfalls.md §G)。
"""
from typing import TYPE_CHECKING, Any, Callable, Optional
from torch import nn
from msmodelslim.model.base import BaseModelAdapter
from msmodelslim.model.interface_hub import (
FA3QuantAdapterInterface, OnlineQuaRotInterface, IterSmoothInterface,
)
if TYPE_CHECKING:
from .dual_expert_base_model_adapter import MyDualExpertBaseAdapter
class MyExpertSubAdapter(
BaseModelAdapter,
OnlineQuaRotInterface, # P7: 显式继承
FA3QuantAdapterInterface, # P7
IterSmoothInterface, # P7
):
def __init__(self, parent: "MyDualExpertBaseAdapter", expert_name: str):
self._parent = parent
self.expert_name = expert_name
self._module: Optional[nn.Module] = None
def bind_module(self, module: nn.Module): self._module = module
def __getattr__(self, item): return getattr(self._parent, item) # 默认代理其它方法
def quantization_context(self):
return self._parent._quantization_context_with_no_sync(self._module)
def generate_model_forward(self, model, inputs):
return self._parent.generate_model_forward(model, inputs)
def generate_model_visit(self, model):
return self._parent.generate_model_visit(model)
def enable_kv_cache(self, model, need_kv_cache):
return self._parent.enable_kv_cache(model, need_kv_cache)
def get_online_rotation_configs(self, model=None):
return self._parent.get_online_rotation_configs(model or self._module)
def inject_fa3_placeholders(self, root_name, root_module, should_inject):
return self._parent.inject_fa3_placeholders(root_name, root_module, should_inject)
def get_adapter_config_for_subgraph(self):
return self._parent.get_adapter_config_for_subgraph(self._module.num_layers)
```
---
## 文件 3:场景子类骨架(每个 scene 一份)
`dual_expert_scene_model_adapter.py`(参考 Wan2.2 t2v/model_adapter)
```python
"""场景子类骨架(T2V / I2V / TI2V / 任一具体场景)。
每个具体场景复制此骨架 → 改 scene_task / InferenceConfig / validate_calib_samples /
init_model / quantization_context / _generate_video / _build_<id>_pipeline。
"""
from typing import Any, Dict, Literal, Optional
import torch.nn as nn
from pydantic import BaseModel, ConfigDict, field_validator, model_validator
from msmodelslim.core.const import DeviceType
from msmodelslim.infra.dataset_loader.vlm_dataset_loader import VlmCalibSample
from msmodelslim.utils.exception import SchemaValidateError
from ..dual_expert_base_model_adapter import MyDualExpertBaseAdapter
class MySceneModelAdapter(MyDualExpertBaseAdapter):
"""<scene_task> 场景适配器骨架。"""
scene_task = "<scene_task>" # 子类固定(如 "t2v-A14B")
DUAL_EXPERT_SCENE_TASKS = frozenset({"<scene_task>"}) # 标明双专家
class MySceneInferenceConfig(BaseModel):
model_config = ConfigDict(extra="forbid")
# 按目标 parse_args 字段重写
size: Optional[str] = "<default_size>"
sample_steps: Optional[int] = <default_steps>
sample_guide_scale: Optional[float] = None
base_seed: Optional[int] = None
task: Optional[str] = "<scene_task>" # I9: 必须含 task 与 scene_task 一致
@field_validator("sample_guide_scale", mode="before")
@classmethod
def _reject_non_cli_guide_scale(cls, value):
if isinstance(value, (tuple, list)):
raise ValueError("sample_guide_scale must be a single float")
return value
@model_validator(mode="before")
@classmethod
def _reject_mismatched_task(cls, data): # I9: 拒绝跨场景串扰
expected = "<scene_task>"
if isinstance(data, dict) and data.get("task") not in (None, expected):
raise ValueError(f"task {data.get('task')!r} does not match {expected!r}")
return data
def get_inference_config_class(self): return self.MySceneInferenceConfig
def validate_calib_samples(self, samples): # J: 场景规则
for idx, s in enumerate(samples):
if not (isinstance(s.text, str) and s.text.strip()):
raise SchemaValidateError(f"<scene_task> sample[{idx}] requires non-empty text")
if s.image is not None: # T2V 禁图;I2V 强图 / TI2V 可选
raise SchemaValidateError(f"<scene_task> sample[{idx}] must not include image")
return samples
def init_model(self, device=DeviceType.NPU) -> Dict[str, nn.Module]:
_ = device
self._load_pipeline()
experts = {"low_noise_model": self.low_noise_model, "high_noise_model": self.high_noise_model}
self._bind_expert_sub_adapters(experts) # P6
return experts
def quantization_context(self): # H: 子类直接 return 基类 ExitStack 上下文
return self._quantization_context_with_no_sync(self.low_noise_model, self.high_noise_model)
def _generate_video(self, prompt, image_path, inference_config):
# 子类调目标 pipeline(如 wan_t2v.generate / hunyuan_video.predict / ...)
self.<scene_pipeline>.generate(prompt, ...)
def _build_<id>_pipeline(self, args, cfg, device, rank):
# 子类创建目标 pipeline 实例
self.<scene_pipeline> = <ScenePipelineClass>(...)
self.low_noise_model = self.<scene_pipeline>.low_noise_model
self.high_noise_model = self.<scene_pipeline>.high_noise_model
self._setup_<id>_dit_runtime(args, self.low_noise_model, self.high_noise_model)
```
---
## 落地步骤(任何新多模态生成模型)
1. 复制对应骨架 → `msmodelslim/model/<model_id>/`;
2. 替换 `<...>` 占位符为目标模型字段(按 `references/dit/architecture_patterns.md §1.2` 列出的方法识别 parse_args 与 block_keyword);
3. 在 `__init__.py` 导出基类 + 场景子类 + 子适配器;
4. 在 `config.ini` 注册新 `model_type` 与 `[ModelAdapterEntryPoints]`;
5. 跑 `bash install.sh`;
6. 走 `msmodelslim-adapter-verification` 四步验证(DiT 必加 `--skip-random-model`);
7. 落盘前先过 `references/dit/pitfalls.md` 的 12 类通用陷阱。
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |