All articles

Deep dives

Running Stable Diffusion on an iPhone: What It Takes

Memory budgets, quantization, tiled decoding, and thermal limits: the engineering behind fitting a diffusion model into a phone.

By · · Updated · 10 min read

Running Stable Diffusion on an iPhone requires fitting its weights and working data into a limited memory budget. If an app exceeds that budget, iOS can terminate it. Prepared model packages and careful memory handling make local generation possible, but actual speed and reliability depend on the device, model, and task.

Getting there takes a stack of techniques. This article covers what the model is made of, how storing its numbers more coarsely shrinks it, how careful loading and piece-by-piece decoding keep it inside an iPhone’s memory budget, and where the physical limits, RAM and heat, still bite.

Stable Diffusion is three networks

A Stable Diffusion pipeline is three separate neural networks that run one after another.

  • The text encoder turns your prompt into a grid of numbers that captures its meaning. (That grid is called an embedding.) It runs once per generation and finishes in a fraction of a second.
  • The diffusion backbone is the heavyweight. In classic Stable Diffusion this part is a UNet. It starts with a canvas of random noise plus the numbers from your prompt. At every step it estimates how much noise the whole canvas is carrying, and the sampler driving it subtracts a scheduled fraction of that estimate. Repeat, and an image matching your prompt gradually emerges. (Each of those steps is called denoising.) Nearly all of the compute time lives here.
  • The VAE decoder turns the backbone’s output into pixels. The backbone does its work on a small, compressed sketch of the picture rather than a full-resolution image. (That sketch is called a latent.) Working small is a large part of why diffusion is feasible at all. The decoder handles the final enlargement to real pixels.

The sizes are lopsided. The backbone holds the large majority of the learned numbers, so it is most of the download and most of the memory footprint. The text encoder and the decoder are small by comparison. SDXL-class models raise the stakes on both ends. They use a bigger backbone for sharper detail, and they encode text more heavily. That is why an SDXL Turbo package is roughly half again the size of a base Turbo package.

Splitting the model this way matters on a phone. The three parts have very different sizes and run at different times. They never all need to be working at the same instant. Every memory strategy that follows exploits that fact.

Quantization: the 75% haircut

A model’s weights are the numbers it learned during training, and they are normally stored as 16-bit decimals (floating-point numbers). A backbone with a billion parameters therefore costs about 2 GB before you have generated anything. SDXL-class backbones are larger still. The fix is to store those numbers more coarsely. Each small block of weights becomes a set of compact whole numbers, typically 8 or 4 bits each. One scale factor per block brings them back to size. (The technical name for this is quantization.) Going from 16-bit to 4-bit cuts the weight storage by 75%. Diffusion backbones tolerate it well. The distribution of outputs shifts slightly, but side-by-side images are usually hard to tell apart.

A prepared, shrunken Stable Diffusion Turbo package lands around 2 GB on disk, and an SDXL Turbo package around 3 GB. Those are download sizes; peak RAM must be measured separately for the pipeline and device. How the encoding works, and where it starts to visibly damage images, is covered in more depth in our article on quantization.

The memory ceiling, and staged loading

The constraint is RAM. iOS gives each app a hard memory ceiling well below the phone’s total RAM. It does not park inactive memory on disk the way a desktop does, and it kills any app that crosses the line. Go over the cap during a generation and the app disappears. Why RAM tiers decide which models a given iPhone can run is the subject of our article on iPhone RAM and AI models. The available headroom differs between devices; a model’s supported tier depends on its complete implementation.

The first defense is staged loading. The three networks run one after another, so they do not all need to sit in memory at once. The text encoder can run and then be released before the backbone starts its steps. The decoder only needs memory at the end. An app that loads everything up front pays for all three networks at its peak. An app that loads and frees parts as the pipeline advances pays something much closer to the cost of its largest single part. On a 6 GB phone, that difference is the difference between running and being terminated.

Staged loading has a cost. Pulling a component off flash storage takes time. A pipeline that aggressively frees and reloads everything trades speed for headroom. The engineering judgment is in choosing what stays in memory and what cycles in and out at the edges of the run. The backbone stays, because it runs many times over. Editing modes raise the stakes further. One mode makes a new image from a photo you supply (img2img). Another repaints a single region of an existing image (inpainting). Between them they carry extra cargo through the pipeline alongside everything else: a source photo, and for inpainting a mask as well. The budget has to hold with that on board too.

Tiled VAE decoding: taming the final spike

The step that stresses memory most is the last one. Turning the compressed sketch into a full-resolution image inflates a small block of numbers into a large one. The decoder’s layers also produce big intermediate results at full image resolution. (Those layers are called convolutional, and the intermediate results are activations.) Done in one pass, this brief spike can exceed the memory used by the entire denoising loop. And it arrives at the moment the app is closest to its ceiling.

The fix is tiled decoding. Split the sketch into overlapping tiles, decode each tile separately, then blend the results back into one image. Peak memory now scales with the tile size rather than the image size. That flattens the spike to a fraction of its naive height. The cost is a little extra compute for the overlapping regions, plus some care at the seams. It is a good trade when the alternative is termination.

Speed, steps, and thermals

Compute is the other half of the problem. Classic Stable Diffusion needed 20 to 50 denoising steps per image. At phone speeds that adds up to minutes per image. Turbo models change the arithmetic. They are trained to copy a slower model’s results in far fewer steps. (That training process is called distillation.) A turbo model produces usable images in one to four steps. They also drop the technique that normally makes a prompt bite. That technique has the model guess twice on every step, once with your words in mind and once ignoring them completely. (The technical name is classifier-free guidance, or CFG.) A turbo model guesses once per step instead of twice. Together those two savings cut the work by roughly a factor of ten, before any hardware optimization happens. Skipping the guidance has a visible side effect. Negative prompts depend on it, so on a turbo model the negative prompt field does nothing. Style and constraints belong in the positive prompt.

Sustained generation can warm a phone enough for the system to reduce performance. Loading, the chosen quality, and other activity also affect timing. Compare several runs on your own device, and pause if it becomes hot. A brief successful generation does not establish sustained performance.

How Onhand puts this together

Onhand includes prepared Turbo and Detailed Images packages of about 2 GB and 3 GB respectively, plus additional model families and Core ML style packs. These are download sizes, not peak RAM measurements. MLX models use the GPU; Core ML packages use their configured compute route.

Staged loading and tiled decoding reduce memory pressure in the pipelines that support them. They do not guarantee that every operation succeeds under every device condition. Download a compatible model and try the workflow you need. Prompts and images stay local during generation; purchases, analytics, and model delivery are described in the privacy policy.

Other apps make these trade-offs differently: Onhand vs PhoneDiffusion compares Onhand with an app that runs Stable Diffusion 1.5 and SDXL on supported devices. What you need for offline use is on the offline AI image generator for iPhone page.

Sources

Primary sources for the claims above, so you can check them rather than take our word for it.