wshobson/agents / plugins/llm-finetuning/skills/lora-qlora-recipes
lora-qlora-recipes
VerifiedConfigure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.5 KB |
| references/hyperparameters.md | markdown | 5.7 KB |
| references/unsloth-trl-mapping.md | markdown | 9.6 KB |
references/hyperparameters.md
View on GitHubLast verified: 2026-07-13
# LoRA/QLoRA Hyperparameter Tables
Full tables and a complete worked config backing
the summary in `SKILL.md`. Base models are never
named here — every example is labeled by size
class only; see `finetuning-method-selection`'s
`references/model-catalog.md` for which actual
model to use at a given size class.
## Rank and Alpha by Task Type
`lora_alpha = 2 * r` in every row — derive alpha
from rank, don't set it independently.
| Task type | Rank (`r`) | `lora_alpha` | Notes |
|---|---|---|---|
| RL adapters (GRPO/RLVR) | 1–32 | 2–64 | Lower end (1–8) is common for adapters on top of an already-capable base. |
| General SFT default | 16–32 | 32–64 | Starting point absent a specific reason to go higher or lower. |
| SFT at scale (large, diverse instruction sets) | up to ~256 | up to ~512 | Only justified when the dataset is large and diverse enough to use the extra capacity — see rsLoRA note below before defaulting here. |
## Learning Rate by Method
LoRA/QLoRA learning rates run roughly **10x** the
equivalent full-fine-tune LR — this is the single
most common misconfiguration when porting a full-
FT config to LoRA (leaving the LR unchanged
under-trains the adapter).
| Method | LR range | Use when |
|---|---|---|
| QLoRA (standard) | **2e-4** | Default starting point for QLoRA SFT. |
| LoRA, conservative | 1e-4 | Larger base model, higher rank, or a run that showed instability at 2e-4. |
| LoRA, very conservative | 5e-5 | Continuing a run, fine-grained behavior adjustment, or a base model that's already close to the target behavior. |
Treat these as starting points to sweep around,
not fixed constants — but start here rather than
porting a full-FT LR unchanged.
## rsLoRA Note
Rank-stabilized LoRA (rsLoRA) rescales the
adapter update by `alpha / sqrt(r)` instead of
`alpha / r`. It's **optional**, and only worth
turning on at **r ≥ 32** — below that rank, the
standard scaling (`alpha / r`) is stable enough
that rsLoRA doesn't change outcomes meaningfully.
If the SFT-at-scale row (rank up to ~256) is in
play, turn rsLoRA on; for the general-default or
RL rows, leave it off unless a specific
instability shows up.
## Batch and Packing Interactions
- Keep **effective batch size under 32** — the
reference recipe was validated at that scale.
Effective batch is `per_device_batch_size *
gradient_accumulation_steps * num_devices`; a
multi-GPU or high-accumulation setup can cross
32 without the per-device batch size looking
large, so compute the product, not just the
per-device number.
- Packing multiple short examples into one
sequence changes the effective batch's token
composition, not just its example count — a
packed batch of 8 sequences is not equivalent
to an unpacked batch of 8 short examples.
Apply the chat template before packing, not
after, and spot-check a handful of decoded
packed sequences before trusting the pipeline.
- Gradient checkpointing (`use_gradient_checkpointing="unsloth"`)
and packing both trade compute for memory
independently — enabling both is normal for a
memory-constrained run, not redundant.
## Worked Config: Unsloth `FastLanguageModel` + `SFTConfig`
A complete, internally consistent config at the
general-default rank (`r=32`), QLoRA, standard
LR. Swap `BASE_MODEL` for an actual checkpoint
from the model catalog before running.
```python
from unsloth import FastLanguageModel
from trl import SFTConfig, SFTTrainer
BASE_MODEL = "<from model catalog>" # size class + task decide this, not this file
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=BASE_MODEL,
max_seq_length=2048,
dtype=None, # auto-detect bf16/fp16 by hardware
load_in_4bit=True, # QLoRA path — set False for bf16 LoRA
)
target_modules = [
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
]
model = FastLanguageModel.get_peft_model(
model,
r=32,
target_modules=target_modules,
lora_alpha=64, # 2 * r
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth",
random_state=3407,
use_rslora=False, # r=32 threshold — leave disabled here unless instability is observed
)
import torch
# Check hardware BF16 support before forcing it — see SKILL.md
# Failure Modes. Training in fp16 on hardware without solid BF16
# support is a known source of loss spikes and silent divergence,
# so this is a hard prerequisite, not a config style choice.
if not torch.cuda.is_bf16_supported():
raise RuntimeError(
"This GPU does not support BF16 — do not fall back to "
"fp16=True as if it were equivalent; pick hardware with "
"BF16 support instead (see SKILL.md Failure Modes)."
)
training_args = SFTConfig(
output_dir="./outputs",
max_length=2048,
dataset_text_field="text",
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch 16 (single device) — stays under 32
learning_rate=2e-4, # QLoRA standard
bf16=True, # gated above — never fp16, see SKILL.md Failure Modes
optim="adamw_8bit",
num_train_epochs=3,
logging_steps=10,
seed=3407,
)
trainer = SFTTrainer(
model=model,
processing_class=tokenizer, # current TRL — not tokenizer=
train_dataset=train_dataset,
args=training_args,
)
trainer.train()
```
This block is internally consistent: `r=32` →
`lora_alpha=64` (2x rule), `load_in_4bit=True` →
`learning_rate=2e-4` (QLoRA standard LR),
`bf16=True` (never fp16), and effective batch
`4 * 4 = 16` (under the 32 ceiling). Changing any
one of these — rank, quantization, or batch
shape — should trigger rechecking the others
against the tables above rather than editing it
in isolation.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirements | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |