Quantization: How Billion-Parameter Models Fit on a Phone

A diffusion model is a huge list of numbers. Billions of them, all learned during training. (The technical name for these numbers is weights.) Stored the way training leaves them, they are far too big for a phone. A billion 16-bit weights takes 2 GB before the model has done anything at all. One compression trick changes that. (The technical name is quantization.) It is the single biggest reason billion-parameter models fit on a phone at all.

The idea is to store each number using fewer bits. Going from 16-bit weights to 4-bit weights cuts a model’s size by 75%. Done carefully, it costs little quality. The word “carefully” stands for the block-by-block bookkeeping that makes low-bit storage survivable. This article covers that bookkeeping and where the trick visibly breaks.

What a weight is, and why 16 bits

Every connection in a neural network has a strength, learned during training. That strength is a weight. These are ordinary decimal numbers, mostly small ones like 0.0231 or −1.407. Training usually produces them as 16-bit floating-point values. That is two bytes each, with enough precision that nobody worries about rounding during training itself. Multiply two bytes by a few billion parameters and you get the multi-gigabyte downloads that make AI models so large.

The whole technique rests on one observation. Running a trained model does not need anything like that precision. (The technical name for running a trained model is inference.) A network’s output is the sum of billions of tiny contributions, and small rounding errors in individual weights mostly wash out in the total. So the weights can be stored far more coarsely, as long as the coarseness is managed.

Group-wise quantization: scales per block

The obvious approach fails immediately. Round every weight in the model to one shared grid of, say, 16 levels, and most of them are ruined. Weights in different layers live at wildly different magnitudes. A grid coarse enough to cover the largest weights would flatten the small ones to zero.

The fix is to compress in small batches instead. Split the weights into small blocks, each holding on the order of tens of consecutive values. Give every block its own tiny ruler: a scale factor, often with an offset, stored at full precision. Inside a block, each weight is recorded as a small whole number measured against that ruler. With 4-bit quantization there are 16 levels to pick from. Rebuilding a weight then takes one multiplication: the whole number times the scale. (The technical name for this is quantizing group-wise.)

Each block’s grid is stretched to fit only its own neighborhood of values, so 16 levels go a long way. A block of small weights gets a fine ruler. A block containing an outlier gets a coarser one, and only its neighbors pay the price. The per-block bookkeeping costs a little extra space, a fraction of a bit per weight. That is why real packages come in slightly above the theoretical minimum. The arithmetic still lands close to the headline number.

What gets compressed here is the weights, the stored numbers the model learned during training. The values the model computes while it runs (its activations) are produced fresh each generation and are usually kept at higher precision, so quality is not eroded twice. Unpacking is cheap too. A GPU rebuilds weights from whole numbers and scales on the fly. So a quantized model is smaller to run as well as to store. Moving fewer bytes through memory is itself a speedup on hardware that is hungry for bandwidth.

Where it breaks

This is lossy compression. At 8 bits, diffusion model output is usually indistinguishable from the original. At 4 bits, careful block-by-block schemes still hold up well for most images. But the error is no longer invisible everywhere. It shows up first in the places that need many weights to be precisely right at once.

Aggressive compression can damage fine detail, prompt following, or color, but the outcome depends on the architecture and method. There is no universal 4-bit boundary between good and bad output. Some components may need higher precision than others. Evaluate the actual package rather than judging it by bit width alone.

Why apps ship pre-quantized packages

The conversion cannot happen on the phone, because it is the one moment when both versions of the model exist at once. You have to read the full-precision weights, all multi-gigabytes of them, while writing the compressed copy. The memory needed during conversion is larger than what the finished model needs at runtime. On a desktop that is a few busy minutes. On an iPhone it is a crash, because iOS caps each app’s memory well below total RAM and terminates offenders, and there is no swap file to absorb the spike. Converting on the phone would also mean downloading the large original over the network only to throw most of those bytes away.

So the conversion happens once, ahead of time, on capable hardware. The phone downloads only the result: a prepared package with the compression scheme, the block scales, and the precision choices already baked in and validated. The phone only runs the compressed model. The same RAM ceiling that forces this choice shapes everything else about on-device generation, which is covered in why RAM decides which models your iPhone can run.

How Onhand handles quantization

Onhand downloads prepared packages, so you do not convert model weights on the phone. Its library includes several compression formats and both MLX and Core ML execution routes. Turbo is about 2.05 GB and Detailed Images about 2.99 GB; other packages differ substantially.

The storage reduction is not a prediction of the same reduction in peak RAM or generation time. Components may remain at higher precision, and decoding needs temporary memory. Compare a package’s results on your own task before judging its quality from its bit width. More on runtime memory is in the Stable Diffusion on iPhone guide.

Some apps let you pick the compression yourself: Onhand vs Off Grid AI covers an app whose documentation lists both palettized (about 1 GB) and full-precision (about 4 GB) image models.

Sources

Primary sources for the claims above, so you can check them rather than take our word for it.

• Apple, MLX documentation. The array framework used to run these quantized weights on Apple silicon.
• Stability AI, SD Turbo model card. Parameter counts behind the size arithmetic in this post.