← Previous Chapter | Table of Contents | Next Chapter →

Chapter 12: Knowledge Distillation and Model Merging¶

Compressing capabilities and synthesizing specialized checkpoints can be achieved without full pretraining through two complementary post-training paradigms: cross-model knowledge distillation and weight-space parameter merging.

12.1 Knowledge Distillation (KD)¶

(Hinton et al., 2015)

Foundational Mechanism: Transfers the dark knowledge (inter-class logit distributions and entropy structures) learned by a high-capacity teacher network to a lightweight student model.

\[\mathcal{L}_{\text{KD}} = \alpha \mathcal{L}_{\text{CE}}(P_S, y) + (1 - \alpha) T^2 D_{\text{KL}}\left( \sigma\left(\frac{z_T}{T}\right) \,\Big\|\, \sigma\left(\frac{z_S}{T}\right) \right)\]

where $z_T$ and $z_S$ represent teacher and student unnormalized logits, $T$ denotes the softmax temperature scaling factor, and $\alpha$ balances hard ground-truth supervision against soft distillation targets.

12.1.1 Generative Distillation Paradigms for LLMs¶

Paradigm	Operational Mechanism	Key Benefits	Representative Examples
Logit-Level Distillation (Offline)	Student matches teacher token distribution over fixed corpora	Captures full dark-knowledge entropy	Classic KD, MiniLLM (Gu et al., 2023)
On-Policy Generalized KD (GKD)	Student samples rollouts; teacher scores student distribution	Mitigates out-of-distribution exposure bias	GKD (Agarwal et al., 2024)
Synthetic Sequence Distillation	Teacher generates demonstrations used as SFT training targets	Simple pipeline, zero logit extraction	Alpaca, Vicuna, UltraFeedback
Reasoning Trace Distillation	Teacher generates step-by-step reasoning tokens ($<\text{think}>\dots</\text{think}>$)	Transfers complex reasoning without RLVR	DeepSeek-R1 Distilled LLaMA / Qwen

The DeepSeek-R1 Distillation Impact: DeepSeek demonstrated that fine-tuning compact foundation models (1.5B to 70B parameters) directly on 800,000 reasoning rollouts distilled from DeepSeek-R1 (671B) yields reasoning benchmarks competitive with models orders of magnitude larger, bypassing complex reinforcement learning loops for the student.

12.2 Weight-Space Model Merging¶

Model merging combines orthogonal capabilities from multiple distinct fine-tuned checkpoints directly in parameter space without requiring gradient optimization or pretraining compute.

12.2.1 Core Merging Algorithmic Taxonomy¶

Linear Task Arithmetic: Calculates task delta vectors relative to the base foundation model: $$W_{\text{merged}} = W_{\text{base}} + \sum_{i=1}^M \lambda_i (W_i - W_{\text{base}})$$

Spherical Linear Interpolation (SLERP): Interpolates between weight vectors along the high-dimensional hypersphere surface, preserving tensor geometric norms and directionality.
TIES-Merging (Trimming, Resolving Signs, Electing) (Yadav et al., 2023):
1. Trim: Retains only top-$k\%$ magnitude parameter updates, setting redundant parameters to zero.
2. Elect Sign: Resolves conflicting parameter directions via majority sign voting across models.
3. Disjoint Merge: Averages only parameter deltas that align with the elected consensus direction.
DARE (Drop And REscale) (Yu et al., 2024): Stochastically masks up to 90-99% of delta parameters to zero and rescales surviving parameters by $\frac{1}{1-p}$, virtually eliminating cross-model interference.
Model Soups (Wortsman et al., 2022): Uniformly averages weight checkpoints fine-tuned under varying learning rates and random seeds, consistently improving out-of-distribution robustness.

12.2.2 Mergekit Toolchain Configuration¶

Production toolkit: arcee-ai/mergekit (the de facto industry standard model merging engine).

# mergekit configuration: DARE-TIES Multi-Model Fusion
merge_method: dare_ties
base_model: meta-llama/Meta-Llama-3-8B
models:
  - model: meta-llama/Meta-Llama-3-8B-Instruct
    parameters:
      weight: 0.4
      density: 0.5
  - model: wizardlm/WizardMath-7B-V1.1
    parameters:
      weight: 0.3
      density: 0.5
  - model: deepseek-ai/deepseek-coder-7b-instruct
    parameters:
      weight: 0.3
      density: 0.5
dtype: bfloat16

Production Utility: Blending general instruction, mathematical reasoning, and coding checkpoints via DARE-TIES produces hybrid foundation models excelling simultaneously across multiple domains with zero additional GPU training hours.

Key Papers¶

Hinton et al. (2015): Distilling the Knowledge in a Neural Network: Foundational knowledge distillation paper.
Sanh et al. (2019): DistilBERT: A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter: Architectural distillation baseline.
Gu et al. (2023): Knowledge Distillation of Large Language Models (MiniLLM): Reverse KL divergence for generative LLM distillation.
Wortsman et al. (2022): Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time: Foundational weight averaging methodology.
Yadav et al. (2023): Resolving Interference When Merging Models (TIES-Merging): Sign resolution and pruning framework.
Yu et al. (2024): Language Models are Super Mario: Absorbing Abilities from Homologous Models with DARE: Extreme delta parameter pruning for model merging.

Exercises¶

Synthetic Distillation Experiment: Use DeepSeek-R1 to synthesize 10,000 reasoning solutions for GSM8K math problems; fine-tune a 1.5B student model on these trajectories and benchmark accuracy gains over standard direct SFT.
Model Soup Robustness Test: Fine-tune three instances of a 1.5B model on identical instruction sets using three distinct random seeds; construct a Model Soup by averaging weights and evaluate whether the merged ensemble outperforms individual runs.
DARE vs. TIES Empirical Comparison: Use mergekit to fuse an instruction-tuned model with a code-specialized model using SLERP, TIES, and DARE; evaluate cross-domain retention on HumanEval and MT-Bench.