Ascend/msagent / skills/modeling/msmodeling-throughput-optimizer-executor
msmodeling-throughput-optimizer-executor
VerifiedInteractively gather parameters for `python -m cli.inference.throughput_optimizer`, generate a deployment simulation command, explain assumptions, ask for execution confirmation, then run the simulation and summarize the best parallel strategy. Use when the user wants to evaluate a model on one or more hardware profiles, compare hardware targets, compare aggregation versus disaggregation versus PD ratio optimization, choose TP/EP/MOE-DP search ranges, or obtain recommended concurrency, throughput, TTFT, and TPOT results from throughput modeling.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.0 KB |
| references/dialog-flow.md | markdown | 5.4 KB |
| references/throughput-optimizer-params.md | markdown | 4.7 KB |
references/throughput-optimizer-params.md
View on GitHub# Throughput Optimizer Parameters
Use this file when building or validating a command for `python -m cli.inference.throughput_optimizer`.
## Core Inputs
- `model_id`: required positional model identifier.
- `--device`: target hardware profile or profiles. The CLI accepts one or more values with `nargs="+"`; pass multiple profiles as `--device A B C` to enable cross-hardware summaries.
- `--num-devices`: total device count. Required for most planning cases even though the CLI has a default. In multi-hardware runs, this same count applies to every profile.
- `--input-length`: required prompt length.
- `--output-length`: required generated length.
## Hardware Profiles
- Single hardware run: use one `--device` value.
- Cross-hardware comparison: use multiple `--device` values in one command.
- Multiple hardware profiles reuse the same model, input/output lengths, deployment mode, quantization, SLO limits, search space, and `--num-devices`.
- Use separate commands when hardware targets require different device counts, different SLOs, or different search spaces.
- Device profile names are validated by the repository. If validation fails, report the unknown profile names and the valid-profile hint from the CLI error.
- Every selected profile must have a communication grid large enough for the shared `--num-devices`.
## Modes
## Domain Deployment Mode To CLI Mapping
| Domain term | Meaning | CLI mode | Required follow-up |
|---|---|---|---|
| PD混部 | Prefill+Decode combined serving layout | aggregation | none |
| PD聚合 | Prefill+Decode combined serving layout | aggregation | none |
| 聚合部署 | Prefill+Decode combined serving layout | aggregation | none |
| PD分离: phase capability | evaluate Prefill/Decode separately | disaggregation | Prefill only, Decode only, or both |
| PD分离: ratio planning | run P and D separately, then match capacity | PD ratio optimization | P/D devices per instance |
| PD ratio | P/D instance ratio planning | PD ratio optimization | P/D devices per instance |
### Aggregation
Default mode. Do not pass `--disagg` or `--enable-optimize-prefill-decode-ratio`.
Typical use:
- one combined serving instance runs Prefill and Decode
- optimize under TTFT, TPOT, or both
### Disaggregation
Pass `--disagg`.
Interpretation:
- only `--ttft-limits`: run Prefill optimization
- only `--tpot-limits`: run Decode optimization
- both limits: run both phases separately
### PD Ratio Optimization
Pass:
- `--enable-optimize-prefill-decode-ratio`
- `--prefill-devices-per-instance`
- `--decode-devices-per-instance`
Do not combine with `--disagg`.
## Quantization
Recommended defaults to offer, not silently apply:
- `--quantize-linear-action W8A8_DYNAMIC`
- `--quantize-attention-action DISABLED`
Custom linear choices:
- `DISABLED`
- `W8A16_STATIC`
- `W8A8_STATIC`
- `W4A8_STATIC`
- `W8A16_DYNAMIC`
- `W8A8_DYNAMIC`
- `W4A8_DYNAMIC`
- `FP8`
- `MXFP4`
Custom attention choices:
- `DISABLED`
- `INT8`
- `FP8`
If linear is `MXFP4`, optionally add `--mxfp4-group-size <n>`.
## Search Dimensions
- `--tp-sizes`
- `--ep-sizes`
- `--moe-dp-sizes`
Rules:
- if all three are omitted, the CLI falls back to TP-only search
- if a search argument is present with no values, the CLI searches powers of two up to world size
- explicit values must be positive, unique enough after normalization, and must not exceed the relevant shared device count
## Performance Constraints
- `--ttft-limits`: positive float in ms
- `--tpot-limits`: positive float in ms
- `--batch-range`: `[max]` or `[min max]`
- `--max-prefill-tokens`: relevant to aggregation mode and effective input length
- `--serving-cost`: optional cost term
## Advanced Options
- `--compile`
- `--compile-allow-graph-break`
- `--jobs`
- `--reserved-memory-gb`
- `--log-level`
- `--dump-original-results`
- `--prefix-cache-hit-rate`: prefix cache hit rate in `[0, 1)`. Ask whether prefix cache is enabled before adding it.
- `--num-mtp-tokens`: number of MTP tokens. Ask whether MTP is enabled and confirm model support before adding it.
- `--mtp-acceptance-rate`: MTP acceptance-rate assumptions in `[0, 1]`. Label user-provided versus heuristic values.
## Multimodal Inputs
For VL models, ask for:
- `--image-height`
- `--image-width`
## Practical Prompting Rules
When the user is unsure:
- ask whether they want one hardware target or a cross-hardware comparison
- ask for the deployment mode first: PD混部/PD聚合, PD分离 phase capability evaluation, or PD ratio planning
- prefer a narrow first run over an exhaustive run
- recommend TP-only search for dense models
- recommend TP-first search for MoE models, then expand to EP or MOE-DP if the user wants a broader comparison
- explicitly ask whether prefix cache and MTP are enabled before the pre-execution summary
- always confirm before execution
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |