Skip to content

rlaope/oh-my-hermes / .claude/agent-skills/omh-model-compression

omh-model-compression

Verified

[omh] Making a model smaller or cheaper via its weights.

FileKindSize
SKILL.mdskill md9.4 KB
references/compression-method.mdmarkdown14.0 KB

references/compression-method.md

View on GitHub
# Model Compression Method

Load this while planning a compression. OMH prepares; the operator runs every command and eval, and each step stays `prepared` until its output is observed. The commands below were read on the page named under each one on 2026-11-07. These libraries change between releases, so open the page again before running one, and record the tool version you ran.

## 1. Size the target

Weight memory is about parameters x bits per weight / 8: an 8B model needs about 16 GB at 16 bits, 8 GB at 8 bits and 4 GB at 4 bits, plus the quantization scales. Add the KV cache, which grows with context length and the number of concurrent sequences, and the runtime's own overhead. This is arithmetic, not a measurement: record the observed file size and peak memory once a candidate exists.

The runtime that will load the result decides the format. Choosing the engine, and which published format to serve, is `omh-inference-serving`; this skill produces the weights.

| Runtime and hardware | Format to produce | Tool |
| --- | --- | --- |
| vLLM on NVIDIA | INT4 weight-only (W4A16), compute capability above 8.0; FP8, compute capability 8.9 or above, and weight-only on 7.5 and above | llm-compressor |
| llama.cpp on CPU, Apple Silicon or an edge device | a GGUF quantization type such as `Q4_K_M`, optionally with an importance matrix | `convert_hf_to_gguf.py` and `llama-quantize` |
| Transformers in one process, or the base of an adapter fine-tune | 8-bit or 4-bit NF4 applied while the model loads | bitsandbytes |
| Transformers with a GPTQ checkpoint | GPTQ int4 | GPT-QModel through `GPTQConfig` |

## 2. Choose the method, cheapest first

Climb in order and stop at the first rung whose candidate passes the quality gate in section 5.

| Rung | Method | Needs | Use when |
| --- | --- | --- | --- |
| 1 | Weight quantization with no calibration data: FP8 dynamic, bitsandbytes, a GGUF type without an importance matrix | the model and the tool | the format exists for the runtime and the loss is within tolerance |
| 2 | Calibrated post-training quantization: GPTQ, AWQ, a GGUF type with an importance matrix | calibration text and GPU time | rung 1 lost too much, or the runtime wants a calibrated INT4 checkpoint |
| 3 | Quantization-aware training | a training run on task data | calibrated 4-bit still loses too much and training compute is available |
| 4 | Pruning that removes channels, heads or layers, followed by recovery training or distillation | a tool that rewrites the architecture, and a training run | fewer parameters are needed, not only fewer bits per weight |
| 5 | Distillation into a smaller student | a teacher, a student architecture, a training run, and terms that allow training on the teacher's outputs | the target needs a smaller architecture than rungs 1 to 4 can give |

Unstructured sparsity saves memory or time only where the runtime has a kernel for that sparsity pattern; measure it in the runtime that will serve, not in a training framework.

## 3. Commands

### llm-compressor: GPTQ W4A16 for vLLM

Source: https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/int4/

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

# Configure the quantization algorithms
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])

# Apply quantization
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
```

The page sets `NUM_CALIBRATION_SAMPLES = 512` and `MAX_SEQUENCE_LENGTH = 2048`, and advises: "Start with 512 samples for calibration data, and increase if accuracy drops". Install with `pip install llmcompressor`.

### llm-compressor: FP8 dynamic for vLLM

Source: https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8/

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Configure the simple PTQ quantization
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

# Apply the quantization algorithm.
oneshot(model=model, recipe=recipe)
```

The page: "we do not need any calibration data for this quantization flow", "FP8 models will run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin.", and "Please use separate environments for vLLM and llm-compressor as they might not work together."

### llm-compressor: AWQ

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/awq/llama_example.py

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform.awq import AWQModifier

recipe = [
    AWQModifier(duo_scaling="both"),
    QuantizationModifier(
        ignore=["lm_head"],
        scheme="W4A16_ASYM",
        targets=["Linear"],
    ),
]
```

The example calls `oneshot` with `dataset=`, `recipe=recipe`, `max_seq_length=512` and `num_calibration_samples=256`, and saves with `model.save_pretrained(SAVE_DIR, save_compressed=True)`. Transformers loads AWQ checkpoints but does not produce them: "Some methods, such as AWQ, AutoRound, AQLM, compressed-tensors, and GGUF, can only be loaded from a checkpoint." (https://huggingface.co/docs/transformers/main/en/quantization/overview)

### llama.cpp: GGUF

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md

```sh
python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M
```

The page: "This process however, may introduce some accuracy loss which is usually measured in Perplexity (ppl) and/or Kullback–Leibler Divergence (kld). This can be minimized by using a suitable imatrix file." Pass the importance matrix to `llama-quantize` with `--imatrix file_name`.

Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md

```sh
./llama-imatrix -m ggml-model-f16.gguf -f calibration-data.txt -ngl 99
```

The output defaults to `imatrix.gguf`; `-o` names another file. The page: "`-m | --model` with a model name and `-f | --file` with a file containing calibration data (such as e.g. `wiki.train.raw`) are mandatory."

### Transformers: GPTQ through GPT-QModel

Source: https://huggingface.co/docs/transformers/main/en/quantization/gptq

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
quantized_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m", device_map="auto", quantization_config=gptq_config)
quantized_model.save_pretrained("opt-125m-gptq")
tokenizer.save_pretrained("opt-125m-gptq")
```

Install with `pip install gptqmodel --no-build-isolation`; the page: "AutoGPTQ is no longer supported in Transformers. Install GPT-QModel instead." `dataset` also takes a list of strings.

### Transformers: bitsandbytes

Source: https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes

```py
import torch
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
```

Add `bnb_4bit_quant_type="nf4"` for NF4, or use `load_in_8bit=True`, and pass the config to `from_pretrained(..., quantization_config=quantization_config)`. The overview page on methods applied while the model loads: "Most of them don't need a calibration dataset." and "[GPTQ](./gptq) is the exception because it calibrates on a dataset while the model loads." (https://huggingface.co/docs/transformers/main/en/quantization/overview) The page: "8 and 4-bit training is only supported for training *extra* parameters."

### torchao: quantization-aware training

Source: https://docs.pytorch.org/ao/main/workflows/qat.html

```python
from torchao.quantization import quantize_, Int4WeightOnlyConfig
from torchao.quantization.qat import QATConfig

base_config = Int4WeightOnlyConfig(group_size=32)
quantize_(model, QATConfig(base_config, step="prepare"))
# train
train_loop(model)
quantize_(model, QATConfig(base_config, step="convert"))
```

The page: "Between these two steps, training can proceed exactly as before."

### PyTorch and NVIDIA Model Optimizer: pruning

Source: https://docs.pytorch.org/docs/2.15/generated/torch.nn.utils.prune.l1_unstructured.html

```python
torch.nn.utils.prune.l1_unstructured(module, name, amount, importance_scores=None)
torch.nn.utils.prune.ln_structured(module, name, amount, n, dim, importance_scores=None)
torch.nn.utils.prune.remove(module, name)
```

Both kinds work by masking. The tutorial (https://docs.pytorch.org/tutorials/intermediate/pruning_tutorial.html): "The pruning mask generated by the pruning technique selected above is saved as a module buffer named weight_mask". On structured pruning it says "this will zero out all the connections corresponding to 50% (3 out of 6) of the channels", and of `remove`: "It simply makes it permanent, instead, by reassigning the parameter weight to the model parameters, in its pruned version." The tensor keeps its shape, so the saved file is no smaller and the model has as many parameters as before. A smaller model needs a tool that rewrites the architecture, as Model Optimizer's Minitron does below, or changing the layers by hand.

Source: https://nvidia.github.io/Model-Optimizer/guides/3_pruning.html

The guide lists three modes for `mtp.prune`:
- `fastnas`: "A pruning method recommended for Computer Vision models." Its `mtp.prune` example is this mode on a ResNet-50, so do not copy it for a language model.
- `mcore_minitron`: "A pruning method developed by NVIDIA Research for pruning GPT, Mamba and Hybrid Transformer Mamba models", "in NVIDIA Megatron-Bridge or Megatron-LM framework."
- `puzzletron`: "An advanced LLM/VLM pruning method by NVIDIA using Mixed Integer Programming (MIP) based NAS search algorithm."

For Minitron and Puzzletron the guide names the Pruning README as the canonical reference.

Source: https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/pruning

Minitron "uses the activation magnitudes to prune the embedding hidden size; mlp ffn hidden size; transformer attention heads;" "and number of layers of the model", and "Slices the model weights according to the pruned architecture". The README: "Both frameworks support importing from a Hugging Face pretrained checkpoint." and "After pruning, distillation is required to recover model accuracy."

### TRL: distillation into a student

Source: https://huggingface.co/docs/trl/main/en/gkd_trainer

```python
from trl.experimental.gkd import GKDConfig, GKDTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
teacher_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-1.5B-Instruct")
training_args = GKDConfig(output_dir="gkd-model", per_device_train_batch_size=1)
trainer = GKDTrainer(
    model=model,
    teacher_model=teacher_model,
    args=training_args,
    processing_class=tokenizer,
    # dataset arguments: see the page
)
trainer.train()
```

`lmbda` sets the share of student-generated outputs and `beta` interpolates between forward and reverse KL divergence; the page reports that a high `lmbda` performed better and that the best `beta` depends on the task. The import path is under `trl.experimental`, so check it against the installed version.

## 4. Calibration data

1. Draw it from the task's distribution: the prompts, languages and formats the model will see. The GPTQ page defaults to `c4` and recommends the paper's dataset; for a model with one job, calibrate once on task text and once on the default, and choose between them on a development split, never on the held-out eval the gate reports.
2. Keep it out of the held-out eval, exact and near-duplicate; record the source, the sample count and the sequence length.
3. Start at the tool's documented size (512 samples at 2048 tokens for llm-compressor GPTQ) and raise it when accuracy drops before moving up a rung.

## 5. Quality gate

1. Run the person's held-out task eval on the uncompressed model, with the prompts and decoding settings the product uses. With sampling on, repeat it at least three times and take the spread as the noise floor.
2. Run the same eval on every candidate, in the runtime that will serve it.
3. Check what the task depends on and an average hides: long inputs, output format and JSON validity, tool calls, and every language the product serves.
4. Add a secondary signal: llama.cpp's `llama-perplexity` with `--kl-divergence-base` and `--kl-divergence` against the 16-bit logits (https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md), or a public task through lm-evaluation-harness, as the vLLM INT4 page runs it:

   ```bash
   lm_eval --model vllm \
     --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A16-G128",add_bos_token=true \
     --tasks gsm8k \
     --num_fewshot 5 \
     --limit 250 \
     --batch_size 'auto'
   ```

   The perplexity README: "Note that perplexity is **not** directly comparable between models, especially if they use different tokenizers." A secondary signal never keeps a candidate the task eval rejects.
5. Keep the smallest candidate within the stated tolerance of the baseline. A loss smaller than the baseline's own spread is no difference; a loss larger than the tolerance sends you one rung up or back to a larger format.
6. Stop when a candidate passes, or when every rung that fits the budget failed: then keeping the uncompressed model, or naming the hardware it needs, is the complete answer.

## 6. Record

For the kept candidate and each rejected one: the base model and revision, its license, the method, the tool and version, the recipe or command, the calibration source, count and sequence length, the output hash, the observed size and peak memory, and every eval score beside the baseline's.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall