Distillation, or how a small model inherits a big one's behavior
Most of the cheap, fast models you use are taught by a larger model rather than trained from scratch on raw data. The mechanism is called distillation. It's how the budget tier gets good — and it comes with the teacher's ceiling and blind spots baked in.
By Jackdaw Team
The short version: Most cheap, fast models are trained to imitate a bigger model rather than built from scratch. That's how the budget tier got good — but the small "student" inherits the big "teacher's" ceiling and blind spots, and is strong only on what it was taught.
When you reach for a small, fast, cheap model, you're usually not using something that was trained from scratch to be small. You're using something that was taught by a bigger model. The technique is knowledge distillation, and it's one of the main reasons the budget tier got good enough to be the right default for most high-volume work. It's also a technique with firm limits, and understanding both halves tells you a lot about when a cheap model will quietly disappoint you.
The basic move: a student imitates a teacher
Distillation trains a small "student" model to reproduce the behavior of a large "teacher" model. Instead of (or in addition to) learning from raw data, the student learns from the teacher's outputs — it watches the teacher respond to a huge range of inputs and learns to produce the same responses. The goal isn't for the student to rediscover everything from scratch; it's to compress the teacher's already-learned behavior into a smaller, cheaper network.
The result, when it works, is a model a fraction of the teacher's size that performs close to the teacher on the kinds of tasks it was distilled for, at a fraction of the inference cost. The "mini" and "flash" tiers are, in large part, distillates of their flagship siblings — which is exactly why they can be so cheap and still so capable.
Why imitation transfers so much
The interesting question is why this works as well as it does — why a small model copying a big one's answers ends up genuinely capable rather than a shallow mimic.
Part of the answer is an insight that goes back to the original work on distillation: the teacher's output carries more information than just the answer. When a model classifies an image, it doesn't only say "cat" — internally it assigns probabilities across all options: maybe 70% cat, 20% dog, 8% fox, and so on. Those proportions encode something real about how the categories relate — that a cat is more dog-like than fox-like in the relevant features. Training the student to match the full distribution, not just the top answer, hands it this "dark knowledge" — the teacher's sense of similarity and structure, not just its final picks. The student learns the teacher's function, a much richer target than the bare labels.
For language models specifically, distillation often takes a more direct form: use the large model to generate a vast set of high-quality examples — questions and worked answers, instructions and responses — and train the small model on that generated data. The big model acts as a tireless expert producing perfect training material tailored to exactly the behaviors you want. The student doesn't have to learn from the messy, contradictory open internet; it learns from a clean curriculum written by something that already knows the answers.
The limits, which are the important part
Distillation's constraints are not footnotes; they're the difference between using a cheap model well and getting burned by one.
The student inherits the teacher's ceiling. A distilled model is learning to imitate the teacher, so on the distilled behavior it generally can't exceed the teacher — the teacher's quality is roughly the cap. It also inherits the teacher's mistakes and biases: if the teacher is wrong about something in a consistent way, the student faithfully learns to be wrong the same way. You're compressing one specific model's behavior, mistakes and all.
It's narrow. A student distilled to be excellent at a particular range of tasks is excellent there and falls off faster outside it than a model that learned broadly. Distillation concentrates capability rather than generalizing it. This is often fine — even ideal — when you know your workload, and a trap when you assume the cheap model is a smaller-but-equivalent generalist. It's smaller-but-equivalent on the distribution it was taught, and weaker off it.
Distilling distillates degrades. Train a model on another model's synthetic output, then train a third on the second's output, and so on, and quality erodes — a phenomenon sometimes called model collapse. Each generation loses some of the diversity and edge-case coverage of the original real data, like a photocopy of a photocopy. Synthetic training data from a strong teacher is powerful; an ecosystem of models trained mostly on each other's outputs is a slow drift away from the ground truth that anchored the first one.
A competitive and legal wrinkle
There's a live tension worth naming neutrally. The most capable teachers are often hosted models you access through an API, and you can, in principle, distill from one by collecting its outputs and training your own model on them — effectively transferring a frontier lab's expensive capability into your cheaper model. Providers generally prohibit exactly this in their terms of service, and there have been high-profile disputes over whether one lab's model was trained on another's outputs. It's hard to prove and harder to prevent, and it sits at the center of an ongoing argument about what's defensible when capability can be copied by imitation. You don't need to take a side to understand the dynamic: distillation makes capability leaky, and that leakiness is reshaping how labs think about what they expose.
What this should tell you
The practical lesson is about expectations. When you use a cheap, fast model, you're most likely using a compression of a bigger one's behavior — which is great news, because it means the budget tier is far better than its size alone would suggest, and it's why "use the small model for the big job" is sound advice for most repetitive work. But it also means the cheap model is best understood as the flagship's behavior on a budget, within bounds — strong on what it was taught, with the teacher's ceiling overhead and a sharper drop-off at the edges of its training. Lean on it confidently for the work it was shaped for. Just don't assume it's a tiny generalist that happens to be cheap; assume it's a specialist that learned from a generalist, and test it on the actual edges of what you're going to ask it to do.