Build A Reasoning Model From Scratch 8: Distillation

Sebastian Raschka · 2026-10-17

Build A Reasoning Model From Scratch 8: DistillationSebastian Raschka ▶ Play the video

Why it earned a slot

Sebastian Raschka explores model distillation as a more efficient alternative to reinforcement learning for developing small reasoning models, discussing soft vs. hard distillation and on-policy vs. off-policy distillation.

The short version

Model distillation for efficient reasoning models

Why it matters

Developing small reasoning models can be challenging, and model distillation offers a promising approach to improve efficiency, which is crucial for applications where computational resources are limited.

My take

I think model distillation is a valuable technique for developing small reasoning models, and its potential benefits should be explored further, particularly in comparison to reinforcement learning.

How it connects

• Sebastian Raschka's discussion of model distillation can be connected to the broader topic of efficient machine learning techniques.
• The use of model distillation in reasoning models can be compared to its application in other areas, such as natural language processing.
• The efficiency gains from model distillation can be evaluated on a specific workload to determine its suitability.

Bottom line

Compare the performance of model distillation to reinforcement learning on your own workload to determine the most efficient approach for developing small reasoning models.