kali20gakki/msAgent / skills/modeling/msmodeling-text-generate-executor
msmodeling-text-generate-executor
VerifiedInteractively gather parameters for `python -m cli.inference.text_generate`, generate a concrete text generation simulation command, explain assumptions, ask for execution confirmation, then run the command and summarize performance metrics. Use when the user wants to validate a specific model, hardware profile, batch/query size, prefill or decode scenario, TP/DP/EP/MOE parallel strategy, profiling database, chrome trace, empirical metrics export, or a best row produced by throughput_optimizer.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.9 KB |
| references/dialog-flow.md | markdown | 3.7 KB |
| references/text-generate-params.md | markdown | 4.4 KB |
references/text-generate-params.md
View on GitHub# Text Generate Parameter Guidance
This reference summarizes high-value parameters for `python -m cli.inference.text_generate`.
## Required Or Core Parameters
- `model_id`: Hugging Face model id or local model directory.
- `--device`: target device profile; default is `TEST_DEVICE`, but real planning should specify a concrete profile.
- `--num-devices`: total device/process count; default is `1`.
- `--num-queries`: number of parallel inference queries in one batch; required.
- `--query-length`: new input sequence length in tokens; required.
- `--context-length`: existing context length in tokens; default is `0`.
- `--decode`: enables autoregressive decoding mode; omit for prefill-only simulation.
## Optimization And Quantization
- `--compile`: enables compiled path simulation; the skill adds it by default for realistic compiled-path validation, shows the assumption, and lets the user disable it before execution.
- `--compile-allow-graph-break`: allows graph breaks during compile for dynamic models.
- `--enable-sequence-parallel`: enables sequence parallel graph rewrite during compilation.
- `--quantize-linear-action`: default is `W8A8_DYNAMIC`.
- `--quantize-attention-action`: default is `DISABLED`.
- `--quantize-lmhead`: off by default because LM head quantization can affect accuracy.
- `--mxfp4-group-size`: default is `32`; relevant when linear quantization is `MXFP4`.
## Parallelism
- `--tp-size`: tensor parallel size for the whole model; default is `1`.
- `--dp-size`: data parallel size for the whole model; optional.
- `--ep-size`: expert parallel size; default is `1`.
- `--o-proj-tp-size`, `--o-proj-dp-size`: attention output projection overrides.
- `--mlp-tp-size`, `--mlp-dp-size`: MLP layer overrides.
- `--lmhead-tp-size`, `--lmhead-dp-size`: LM head overrides.
- `--moe-tp-size`, `--moe-dp-size`: expert-layer overrides.
- `--word-embedding-tp`: enables word embedding TP with `col` or `row`.
- `--enable-redundant-experts`: enables redundant experts for relevant MoE layouts.
- `--enable-shared-expert-tp`: enables vLLM-style TP for shared experts.
- `--enable-dispatch-ffn-combine`: enables dispatch/FFN/combine fusion during compilation.
- `--enable-external-shared-experts`: uses external shared experts.
- `--host-external-shared-experts`: hosts external shared experts on the current device.
## Prefix Cache And MTP
- `--prefix-cache-hit-rate`: token-level prefix cache hit-rate approximation in `[0, 1)`.
- `--num-mtp-tokens`: number of MTP tokens; use only for models with MTP capability.
- `--disable-repetition`: preserves original transformer behavior instead of exploiting repetition patterns.
## Multimodal
- `--image-batch-size`: batch size for image processing.
- `--image-height`: input image height.
- `--image-width`: input image width.
If any image option is needed, collect all three values.
## Performance Model And Debug
After the candidate command is generated, always present `--chrome-trace`, `--graph-log-url`, and `--dump-input-shapes` as optional Debug/Trace choices before final execution confirmation.
- `--performance-model analytic`: default roofline-style model.
- `--performance-model profiling`: empirical model backed by profiling CSV data.
- `--profiling-database`: required for profiling mode.
- `--export-empirical-metrics`: developer export path; requires profiling mode.
- `--chrome-trace`: writes a Chrome trace file.
- `--graph-log-url`: dumps compiled graphs when compile is enabled.
- `--dump-input-shapes`: groups table averages by input shapes.
- `--num-hidden-layers-override`: debug-only hidden layer override.
- `--remote-source`: `huggingface` by default; use `modelscope` when requested.
- `--log-level`: defaults to `error`; use `info` or `debug` for troubleshooting.
## Common Templates
Prefill-only analytic validation:
```powershell
python -m cli.inference.text_generate MODEL_ID --device DEVICE --num-devices N --num-queries B --query-length INPUT_TOKENS --context-length 0 --tp-size TP --compile
```
Decode validation:
```powershell
python -m cli.inference.text_generate MODEL_ID --device DEVICE --num-devices N --num-queries B --query-length INPUT_TOKENS --context-length CONTEXT_TOKENS --decode --tp-size TP --compile
```
Profiling validation:
```powershell
python -m cli.inference.text_generate MODEL_ID --device DEVICE --num-devices N --num-queries B --query-length INPUT_TOKENS --performance-model profiling --profiling-database PATH
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |