Quantization, or running a model at 4 bits
A model's weights are normally stored at 16 bits of precision. You can run them at 8, or 4, or fewer — shrinking the model and speeding it up for a small accuracy cost. It's how big models fit on small hardware, and it's nearly everywhere.
By Jackdaw Team
The short version: A model's weights are usually stored at 16 bits of precision. Quantization runs them at 8, 4, or even fewer — which shrinks the model and speeds it up for a small, task-dependent accuracy cost. It's nearly universal, which means "the same model" can behave differently depending on how it was compressed.
A model is, at bottom, a very large pile of numbers — its weights. When people say a model "is 70 billion parameters," they mean it has 70 billion of those numbers. How precisely you store each one turns out to be a dial you can turn, and turning it is one of the most consequential and least visible decisions in how a model actually gets served. That dial is quantization.
Precision is a choice, and the default is 16 bits
Each weight is normally stored as a 16-bit floating-point number — two bytes, enough precision to represent a fine-grained range of values. That precision was useful during training, where tiny adjustments accumulate. But for running a trained model, it's often more than you need. The question quantization asks is: what if we stored and computed those numbers at lower precision — 8 bits, or 4, or fewer?
The payoff is immediate and comes from two directions.
Memory. A model's size in memory is roughly its parameter count times the bytes per parameter. At 16 bits (2 bytes), a 70-billion-parameter model needs about 140 GB just for its weights — more than a single GPU typically has, so you need several. Drop to 4 bits (half a byte) and the same model needs about 35 GB, which fits on one. Quantization is frequently the difference between "needs a cluster" and "runs on one card," which is the difference between feasible and not for a lot of people.
Speed. Running a model is heavily bottlenecked by moving those weights from memory to the compute units — it's bandwidth-bound more than math-bound. Fewer bytes per weight means less data to move, which means faster inference. Smaller and faster at once.
How you squeeze a 16-bit number into 4 bits
The mechanism is, in essence, rounding to a coarser grid. Take a group of weights, find the range they span, and map that range onto the small set of values your low-bit format can represent — at 4 bits, only sixteen distinct values — storing each weight as the nearest grid point plus a shared scaling factor to get back to the right magnitude. You've thrown away the fine detail between grid points, but kept the overall structure.
The thing that makes this hard, and the thing all the clever methods are really about, is outliers. Most weights cluster in a narrow range, but a few are far larger. If you stretch your sixteen grid points to cover those rare large values, the dense middle — where most of the weights live — gets represented coarsely, and quality craters. So good quantization methods treat outliers specially: keeping the unusual values at higher precision, scaling groups of weights separately, or analyzing which weights matter most and protecting them. The well-known techniques in this space are all, underneath, different answers to "how do we round aggressively without letting a handful of extreme values wreck everything."
What you lose
You don't get something for nothing, but the something you lose is often surprisingly small.
- 8-bit is usually near-lossless — most models run at 8-bit with quality differences too small to notice on most tasks. This is close to a free win and extremely common.
- 4-bit is the popular sweet spot, especially for running models locally. There's a measurable quality drop, but with good methods it's modest, and the memory savings are large enough that it's often worth it.
- Below 4 bits, degradation accelerates and you're trading real capability for the last increments of size.
Two important nuances. First, the loss isn't uniform across tasks — quantization tends to hit the hard things hardest. Easy classification or extraction may show no visible difference at 4-bit, while complex multi-step reasoning, long-context performance, or precise factual recall degrade more. A model can look "basically the same" on a casual test and be meaningfully worse on exactly the demanding work you'd want a big model for. Second, you can quantize more than the weights: the KV cache — the memory that grows with context length — can also be quantized, which is a separate lever specifically for making long contexts cheaper.
Why it's everywhere, and why that should make you careful
Inference cost dominates the economics of deploying a model — you pay it on every request, forever — so anything that cuts memory and latency is enormously attractive. Quantization is the cheapest such lever: it requires no retraining (in its common post-training form), it's well understood, and it stacks with everything else. So it's nearly universal. Most served models are quantized to some degree, and models people run on their own hardware are almost always heavily quantized — a "local Llama" is essentially never the full-precision model.
That ubiquity has a consequence worth internalizing: "the same model" is not a fixed thing. A model identifier tells you the weights' origin, not the precision they're being run at. A 4-bit local copy and the full-precision hosted version share a name and don't share behavior, especially on hard tasks. Two providers serving "the same" open model may quantize it differently and get different results. Benchmarks rarely specify the precision they tested. So when you compare models, or when a local model underperforms the hosted one you tried, quantization is often the hidden variable.
The clean way to hold it: precision is a dial between quality and cost, it's turned down far more often than people realize, and most deployments are not running the model as it was trained. If you're choosing between a smaller model at full precision and a larger model quantized down to the same memory footprint, that's a real and non-obvious trade — and the only honest way to settle it is to test both on your own task, because which one wins depends entirely on how much your task leans on the precision that got rounded away.