Jeffallan/claude-skills / skills/fine-tuning-expert
fine-tuning-expert
VerifiedUse when fine-tuning LLMs, training custom models, or adapting foundation models for specific tasks. Invoke for configuring LoRA/QLoRA adapters, preparing JSONL training datasets, setting hyperparameters for fine-tuning runs, adapter training, transfer learning, finetuning with Hugging Face PEFT, OpenAI fine-tuning, instruction tuning, RLHF, DPO, or quantizing and deploying fine-tuned models. Trigger terms include: LoRA, QLoRA, PEFT, finetuning, fine-tuning, adapter tuning, LLM training, model training, custom model.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.6 KB |
| references/dataset-preparation.md | markdown | 16.1 KB |
| references/deployment-optimization.md | markdown | 16.7 KB |
| references/evaluation-metrics.md | markdown | 17.6 KB |
| references/hyperparameter-tuning.md | markdown | 16.2 KB |
| references/lora-peft.md | markdown | 10.8 KB |
references/lora-peft.md
View on GitHub# LoRA and Parameter-Efficient Fine-Tuning
---
## Overview
Parameter-Efficient Fine-Tuning (PEFT) methods train only a small subset of model parameters while keeping the base model frozen. This dramatically reduces memory requirements and enables fine-tuning of large models on consumer hardware.
## When to Use PEFT vs Full Fine-Tuning
| Method | Use When | Avoid When |
|--------|----------|------------|
| **LoRA** | 7B+ models, limited VRAM, need multiple task adapters | Very small models (<1B), need maximum quality |
| **QLoRA** | 13B+ models, single GPU, memory-constrained | High-throughput training, inference speed critical |
| **Full Fine-Tuning** | Small models, abundant compute, maximum performance needed | Large models, limited resources |
| **Prefix Tuning** | Generation tasks, need interpretable soft prompts | Complex reasoning tasks |
| **IA3** | Extreme efficiency needed, inference overhead critical | Tasks needing high adapter capacity |
## LoRA Configuration
```python
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load base model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="flash_attention_2" # Use Flash Attention if available
)
# LoRA configuration for instruction tuning
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank - start with 8-16, increase if underfitting
lora_alpha=32, # Alpha - typically 2x rank
lora_dropout=0.05, # Dropout for regularization
target_modules=[ # Target attention layers
"q_proj", "k_proj", "v_proj", "o_proj", # Attention
"gate_proj", "up_proj", "down_proj" # MLP (optional, increases capacity)
],
bias="none", # "none", "all", or "lora_only"
modules_to_save=None # Modules to train fully (e.g., embed_tokens for new tokens)
)
# Create PEFT model
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 13,631,488 || all params: 8,043,163,648 || trainable%: 0.1695
```
### Rank Selection Guide
```python
def recommend_lora_rank(task_complexity: str, dataset_size: int, model_size_b: float) -> int:
"""
Recommend LoRA rank based on task and resources.
Args:
task_complexity: "simple" (classification), "moderate" (QA), "complex" (creative)
dataset_size: Number of training examples
model_size_b: Model size in billions of parameters
"""
base_rank = {
"simple": 8,
"moderate": 16,
"complex": 32
}[task_complexity]
# Adjust for dataset size
if dataset_size < 1000:
rank = max(4, base_rank // 2) # Reduce rank to prevent overfitting
elif dataset_size > 50000:
rank = min(64, base_rank * 2) # Can support higher rank
else:
rank = base_rank
# Adjust for model size (larger models may need lower rank)
if model_size_b > 30:
rank = max(4, rank // 2)
return rank
# Example usage
rank = recommend_lora_rank("moderate", dataset_size=10000, model_size_b=8)
print(f"Recommended rank: {rank}") # 16
```
## QLoRA Configuration
QLoRA combines 4-bit quantization with LoRA for extreme memory efficiency.
```python
from transformers import BitsAndBytesConfig
import torch
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 for better quality
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True # Nested quantization for more savings
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B",
quantization_config=bnb_config,
device_map="auto",
attn_implementation="flash_attention_2"
)
# Prepare model for kbit training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)
# Apply LoRA
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(model, lora_config)
```
### Memory Comparison
| Model | Full FT | LoRA (r=16) | QLoRA (r=16) |
|-------|---------|-------------|--------------|
| Llama 3.1 8B | ~64 GB | ~18 GB | ~6 GB |
| Llama 3.1 70B | ~560 GB | ~160 GB | ~48 GB |
| Mistral 7B | ~56 GB | ~16 GB | ~5 GB |
## Training with PEFT
```python
from transformers import TrainingArguments, Trainer
from trl import SFTTrainer
training_args = TrainingArguments(
output_dir="./lora-output",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 16
learning_rate=2e-4, # Higher LR for LoRA than full FT
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=10,
save_strategy="steps",
save_steps=100,
evaluation_strategy="steps",
eval_steps=100,
bf16=True,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
optim="paged_adamw_8bit", # Memory-efficient optimizer
max_grad_norm=0.3,
group_by_length=True, # Group similar length sequences
report_to="wandb"
)
# Using TRL's SFTTrainer for instruction tuning
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
max_seq_length=2048,
packing=True, # Pack short sequences for efficiency
dataset_text_field="text"
)
trainer.train()
```
## Target Module Selection
Different architectures have different module names:
```python
# Common target modules by architecture
TARGET_MODULES = {
"llama": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
"mistral": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
"falcon": ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"],
"gpt2": ["c_attn", "c_proj", "c_fc"],
"phi": ["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"],
"qwen2": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
}
def get_target_modules(model_name: str, include_mlp: bool = True) -> list[str]:
"""Get appropriate target modules for a model architecture."""
name_lower = model_name.lower()
for arch, modules in TARGET_MODULES.items():
if arch in name_lower:
if include_mlp:
return modules
# Return only attention modules
attention_keywords = ["q_proj", "k_proj", "v_proj", "o_proj", "query", "key", "value", "attn"]
return [m for m in modules if any(kw in m.lower() for kw in attention_keywords)]
# Default for unknown architectures - inspect model
raise ValueError(f"Unknown architecture: {model_name}. Inspect model.named_modules() to find target modules.")
```
## Adapter Merging Strategies
```python
from peft import PeftModel
# Load base model and adapter
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base_model, "path/to/lora-adapter")
# Method 1: Merge adapter weights into base model
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
# Method 2: Merge multiple adapters (weighted combination)
from peft import add_weighted_adapter
# Load multiple adapters
model = PeftModel.from_pretrained(base_model, "adapter1", adapter_name="adapter1")
model.load_adapter("adapter2", adapter_name="adapter2")
model.load_adapter("adapter3", adapter_name="adapter3")
# Combine with weights
model.add_weighted_adapter(
adapters=["adapter1", "adapter2", "adapter3"],
weights=[0.5, 0.3, 0.2],
adapter_name="combined",
combination_type="linear" # or "svd", "cat"
)
model.set_adapter("combined")
```
## DoRA (Weight-Decomposed LoRA)
DoRA improves on LoRA by decomposing weights into magnitude and direction components.
```python
from peft import LoraConfig
# DoRA configuration
dora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
use_dora=True, # Enable DoRA
task_type=TaskType.CAUSAL_LM
)
# Training is identical to LoRA
model = get_peft_model(model, dora_config)
```
## rsLoRA (Rank-Stabilized LoRA)
Proper scaling for higher ranks:
```python
from peft import LoraConfig
# rsLoRA for high-rank training
rslora_config = LoraConfig(
r=64, # Higher rank
lora_alpha=64,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
use_rslora=True, # Rank-stabilized scaling
task_type=TaskType.CAUSAL_LM
)
```
## Common Issues and Solutions
### Issue: Loss Not Decreasing
```python
# Check 1: Verify adapter is training
for name, param in model.named_parameters():
if param.requires_grad:
print(f"Training: {name}")
# Check 2: Increase rank or alpha
config = LoraConfig(r=32, lora_alpha=64, ...)
# Check 3: Reduce learning rate
training_args = TrainingArguments(learning_rate=1e-4, ...)
```
### Issue: Out of Memory
```python
# Solution 1: Use QLoRA
bnb_config = BitsAndBytesConfig(load_in_4bit=True, ...)
# Solution 2: Enable gradient checkpointing
model.gradient_checkpointing_enable()
# Solution 3: Reduce batch size, increase gradient accumulation
training_args = TrainingArguments(
per_device_train_batch_size=1,
gradient_accumulation_steps=16
)
# Solution 4: Use 8-bit optimizer
training_args = TrainingArguments(optim="paged_adamw_8bit")
```
### Issue: Adapter Not Loading
```python
# Ensure architecture matches
from peft import PeftModel, PeftConfig
# Check adapter config
config = PeftConfig.from_pretrained("path/to/adapter")
print(f"Base model: {config.base_model_name_or_path}")
print(f"Target modules: {config.target_modules}")
# Load with correct base model
base_model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(base_model, "path/to/adapter")
```
## Quick Reference
| Parameter | Typical Range | Effect |
|-----------|---------------|--------|
| `r` (rank) | 4-64 | Adapter capacity; higher = more expressive |
| `lora_alpha` | r to 2*r | Scaling factor; higher = larger updates |
| `lora_dropout` | 0.0-0.1 | Regularization; increase for small datasets |
| `learning_rate` | 1e-5 to 3e-4 | LoRA tolerates higher LR than full FT |
| `target_modules` | attention + MLP | More modules = more capacity + memory |
## Related References
- `hyperparameter-tuning.md` - Learning rate schedules, batch sizes
- `deployment-optimization.md` - Adapter merging, quantization for inference
- `dataset-preparation.md` - Training data formatting
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |