Skip to content

Orchestra-Research/AI-Research-SKILLs / 19-emerging-techniques/model-merging

model-merging

Verified

Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.

FileKindSize
SKILL.mdskill md12.5 KB
references/coefficient-tuning.mdmarkdown14.3 KB
references/evaluation.mdmarkdown11.4 KB
references/examples.mdmarkdown8.6 KB
references/methods.mdmarkdown9.3 KB

references/methods.md

View on GitHub
# Model Merging Methods: Deep Dive

Complete technical guide to model merging algorithms based on research papers.

## Table of Contents
- TIES-Merging Algorithm
- DARE (Drop And REscale)
- Linear Merging
- SLERP
- Task Arithmetic
- Comparison

## TIES-Merging: Resolving Interference

**Paper**: "TIES-Merging: Resolving Interference When Merging Models" (NeurIPS 2023)
**Authors**: Prateek Yadav et al.
**Code**: https://github.com/prateeky2806/ties-merging

### Algorithm Overview

TIES-Merging addresses two major sources of interference:
1. Redundant parameter values
2. Sign disagreement across models

**Three-Step Process**: TRIM, ELECT, MERGE

### Step 1: TRIM (Reset Small Changes)

Remove parameters that changed minimally during fine-tuning.

```python
def trim(task_vector, density=0.2):
    """Keep top-k% parameters by magnitude, reset rest to 0."""
    # Calculate magnitude
    magnitudes = torch.abs(task_vector)

    # Get threshold for top-k%
    k = int(density * task_vector.numel())
    threshold = torch.topk(magnitudes.flatten(), k).values.min()

    # Create mask: keep parameters above threshold
    mask = magnitudes >= threshold

    # Apply mask
    trimmed_vector = task_vector * mask

    return trimmed_vector

# Example
task_vector_1 = finetuned_model_1 - base_model
task_vector_2 = finetuned_model_2 - base_model

trimmed_1 = trim(task_vector_1, density=0.2)  # Keep top 20%
trimmed_2 = trim(task_vector_2, density=0.2)
```

### Step 2: ELECT SIGN (Resolve Conflicts)

When parameters have conflicting signs, elect the dominant sign.

```python
def elect_sign(task_vectors):
    """Resolve sign conflicts across multiple task vectors."""
    # Stack all task vectors
    stacked = torch.stack(task_vectors)  # (num_models, num_params)

    # Count positive vs negative for each parameter
    positive_count = (stacked > 0).sum(dim=0)
    negative_count = (stacked < 0).sum(dim=0)

    # Elect majority sign
    final_sign = torch.where(
        positive_count > negative_count,
        torch.ones_like(stacked[0]),
        -torch.ones_like(stacked[0])
    )

    # Where tie, keep sign from first model
    tie_mask = (positive_count == negative_count)
    final_sign[tie_mask] = torch.sign(stacked[0][tie_mask])

    return final_sign

# Example
task_vectors = [trimmed_1, trimmed_2, trimmed_3]
elected_sign = elect_sign(task_vectors)
```

### Step 3: MERGE (Disjoint Merging)

Merge only parameters that agree with elected sign.

```python
def ties_merge(base_model, task_vectors, density=0.2, lambda_param=1.0):
    """Complete TIES-Merging algorithm."""
    # Step 1: Trim each task vector
    trimmed_vectors = [trim(tv, density) for tv in task_vectors]

    # Step 2: Elect sign
    elected_sign = elect_sign(trimmed_vectors)

    # Step 3: Merge aligned parameters
    merged_task_vector = torch.zeros_like(task_vectors[0])

    for tv in trimmed_vectors:
        # Keep only parameters aligned with elected sign
        aligned_mask = (torch.sign(tv) == elected_sign) | (tv == 0)
        aligned_params = tv * aligned_mask

        # Accumulate
        merged_task_vector += aligned_params

    # Average
    num_models = len(task_vectors)
    merged_task_vector /= num_models

    # Add back to base model
    final_model = base_model + lambda_param * merged_task_vector

    return final_model

# Usage
base = load_model("mistralai/Mistral-7B-v0.1")
model_1 = load_model("WizardLM/WizardMath-7B-V1.1")
model_2 = load_model("teknium/OpenHermes-2.5-Mistral-7B")
model_3 = load_model("NousResearch/Nous-Hermes-2-Mistral-7B-DPO")

task_vectors = [
    model_1 - base,
    model_2 - base,
    model_3 - base
]

merged = ties_merge(base, task_vectors, density=0.5, lambda_param=1.0)
```

### Hyperparameters

**density** (ρ): Fraction of parameters to keep (default: 0.2)
- Lower (0.1-0.3): More aggressive pruning, higher sparsity
- Higher (0.5-0.8): Conservative pruning, denser result

**lambda** (λ): Scaling factor for merged task vector (default: 1.0)
- Lower (<1.0): Less influence from fine-tuned models
- Higher (>1.0): More influence from fine-tuned models

## DARE: Drop And REscale

**Paper**: "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch" (arXiv 2311.03099, 2023)
**Authors**: Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li

### Algorithm

DARE randomly drops delta parameters and rescales remaining ones.

### Mathematical Formulation

Given:
- Base model parameters: θ₀
- Fine-tuned model parameters: θₜ
- Delta parameters: δₜ = θₜ - θ₀

**Step 1: Random Drop**

```
m_t ~ Bernoulli(p)  # Drop mask
δ̃_t = (1 - m_t) ⊙ δ_t  # Element-wise product
```

**Step 2: Rescale**

```
δ̂_t = δ̃_t / (1 - p)  # Rescale to preserve expectation
```

**Final Model**

```
θ̂_t = θ₀ + δ̂_t
```

### Implementation

```python
def dare(base_model, finetuned_model, drop_rate=0.9):
    """DARE: Drop And REscale delta parameters."""
    # Compute delta
    delta = finetuned_model - base_model

    # Random drop mask (Bernoulli)
    drop_mask = torch.bernoulli(torch.full_like(delta, drop_rate))

    # Apply mask (keep 1-p, drop p)
    dropped_delta = delta * (1 - drop_mask)

    # Rescale to preserve expectation
    rescaled_delta = dropped_delta / (1 - drop_rate)

    # Reconstruct model
    result = base_model + rescaled_delta

    return result

# Example
base = load_model("mistralai/Mistral-7B-v0.1")
finetuned = load_model("WizardLM/WizardMath-7B-V1.1")

# Drop 90% of delta parameters
result = dare(base, finetuned, drop_rate=0.9)
```

### DARE + TIES (DARE-TIES)

Combine both methods for best results.

```python
def dare_ties(base_model, finetuned_models, drop_rate=0.9, density=0.5):
    """DARE + TIES-Merging."""
    # Step 1: Apply DARE to each model
    dare_deltas = []
    for model in finetuned_models:
        delta = model - base_model

        # DARE drop
        drop_mask = torch.bernoulli(torch.full_like(delta, drop_rate))
        dropped = delta * (1 - drop_mask)
        rescaled = dropped / (1 - drop_rate)

        dare_deltas.append(rescaled)

    # Step 2: Apply TIES to DARE-processed deltas
    merged = ties_merge(base_model, dare_deltas, density=density)

    return merged
```

### Hyperparameters

**drop_rate** (p): Probability of dropping each parameter (default: 0.9)
- Lower (0.5-0.7): Conservative, keeps more parameters
- Higher (0.9-0.99): Aggressive, maximum sparsity
- Works well even at 0.99 for large models

**Observations**:
- Larger models tolerate higher drop rates
- Delta parameters with small absolute values (<0.002) can be safely dropped
- Performance improves with model size

## Linear Merging (Model Soup)

Simple weighted average.

```python
def linear_merge(models, weights):
    """Weighted average of model parameters."""
    assert len(models) == len(weights)
    assert sum(weights) == 1.0, "Weights should sum to 1"

    merged = sum(w * model for w, model in zip(weights, models))

    return merged

# Example
models = [model_1, model_2, model_3]
weights = [0.4, 0.3, 0.3]
merged = linear_merge(models, weights)
```

## SLERP: Spherical Linear Interpolation

Interpolate along sphere in weight space.

```python
def slerp(model_1, model_2, t=0.5):
    """SLERP between two models."""
    # Flatten parameters
    p1 = torch.cat([p.flatten() for p in model_1.parameters()])
    p2 = torch.cat([p.flatten() for p in model_2.parameters()])

    # Normalize
    p1_norm = p1 / p1.norm()
    p2_norm = p2 / p2.norm()

    # Compute angle
    dot = (p1_norm * p2_norm).sum()
    theta = torch.acos(torch.clamp(dot, -1.0, 1.0))

    # SLERP formula
    if theta < 1e-6:
        # Vectors nearly parallel, use linear interpolation
        result = (1 - t) * p1 + t * p2
    else:
        # Spherical interpolation
        sin_theta = torch.sin(theta)
        result = (torch.sin((1 - t) * theta) / sin_theta) * p1 + \
                 (torch.sin(t * theta) / sin_theta) * p2

    # Reshape back to model
    merged_model = reshape_to_model(result, model_1)

    return merged_model

# Example
merged = slerp(model_1, model_2, t=0.5)  # 50-50 blend
```

## Task Arithmetic

Add task vectors to base model.

```python
def task_arithmetic(base_model, finetuned_models, lambdas):
    """Task arithmetic merging."""
    # Extract task vectors
    task_vectors = [model - base_model for model in finetuned_models]

    # Weighted sum
    combined_vector = sum(λ * tv for λ, tv in zip(lambdas, task_vectors))

    # Add to base
    merged = base_model + combined_vector

    return merged

# Example
base = load_model("mistralai/Mistral-7B-v0.1")
math_model = load_model("WizardLM/WizardMath-7B-V1.1")
code_model = load_model("ajibawa-2023/Code-Mistral-7B")

merged = task_arithmetic(
    base,
    [math_model, code_model],
    lambdas=[0.6, 0.4]
)
```

## Method Comparison

| Method | Pros | Cons | Best For |
|--------|------|------|----------|
| **Linear** | Simple, fast | Basic averaging | 2-3 similar models |
| **SLERP** | Preserves magnitude | Only 2 models | Smooth blending |
| **Task Arithmetic** | Intuitive, flexible | Sign conflicts | Multiple specialized models |
| **TIES** | Resolves conflicts | More complex | Many task-specific models |
| **DARE** | High sparsity | Random variance | Reducing redundancy |
| **DARE-TIES** | Best performance | Most complex | Production (state-of-art) |

## Resources

- **TIES Paper**: https://arxiv.org/abs/2306.01708
- **DARE Paper**: https://arxiv.org/abs/2311.03099
- **mergekit**: https://github.com/arcee-ai/mergekit
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall