All articles

Explainers

Diffusion Models Explained: How AI Images Are Made

A plain-language look at how diffusion models learn to remove noise, turn text into images, and power image generation without the math.

By · · Updated · 8 min read

A diffusion model makes an image by starting from random static and clearing it away a little at a time until a picture that matches your prompt is left. (The technical word for that static is noise.) The network behind it learned a single narrow skill: look at a picture buried under static and work out which part is the static. Many image generators use diffusion, including the Stable Diffusion models discussed here, the big cloud services, and the small models that fit on a phone. Painting dragons, restyling your photos, and matching a written description all come from running that one skill in a loop.

This article walks through how that works, with no math. It covers how the models are trained, what happens between your prompt and the finished picture, and what the knobs in image apps control: seeds, steps, and model choice.

Training: learning to un-ruin pictures

Start with the easy direction, which takes no intelligence at all. Take a photograph and sprinkle a little random static over it. Then add a little more. Then more again. After enough rounds the photo is gone, and nothing is left but static, like a dead TV channel. Ruining pictures this way is easy, which makes it a good training exercise: you can manufacture endless examples for free.

Training runs that exercise backwards. The network sees an image part-way through the ruining. Sometimes it is lightly speckled. Sometimes it is almost pure static. The network has to say which parts are static and which parts are picture. We know exactly what static was added, so we can grade the answer against the truth. Then we nudge the network to do a little better next time. Repeat that across billions of images. The network gets extraordinarily good at telling picture from static, at every level of damage.

To say “that part is noise” the network needs an opinion about the picture underneath. A model that can reliably find the dog under the static has, in effect, learned what dogs look like. All its knowledge of the visual world lives inside that one ability: knowing what does not belong.

Generation: sculpting from static

To make a new image, you hand the model a lie. You give it a canvas of pure random noise, with no picture hidden in it at all. Then you ask the usual question: which part of this is noise? The model has only ever seen noise laid over real images, so it invents a faint answer. Take away this bit of static, it says, and there might be a picture under here. You remove a little of what it points to and ask again. The canvas is now slightly less random, so the next guess is slightly more confident. Each pass firms the image up a bit more: vague masses first, then shapes, then edges and texture.

The old line about sculpture applies: carve away everything that is not the statue. The model never paints. It only removes what it judges does not belong, and the image is whatever survives. Each pass through the loop is called a denoising step. Classic models used 20 to 50 of them. Distilled models, versions retrained to take fewer, larger jumps along the same path, compress that whole journey into a few. What a single step computes, and why more of them is not more intelligence, gets its own article: denoising steps, explained.

Where your prompt comes in

Left alone, the loop above would produce an image. Some plausible scene, of nothing in particular. Your prompt is what aims it. A second network reads your words and turns them into a list of numbers that captures their meaning. (That network is called the text encoder.) In those numbers, “a rusty red pickup truck” lands near actual pictures of rusty red pickup trucks. It can do that because it was trained on hundreds of millions of image-caption pairs. Words and pictures end up described in the same terms.

Those numbers go to the denoising network on every single pass, and the network was trained to let them tip the scales. Asked what sits under the noise, with pickup truck whispering alongside, it settles unclear patches toward truck-shaped answers. The prompt never draws anything itself. Instead it tips thousands of small judgments about what belongs. That is why wording and detail matter so much. It is also why the model still takes liberties with anything you left unsaid.

The compressed canvas: latent space and the VAE

One practical detail separates real Stable Diffusion-style models from the simple story above. The denoising loop does not run on pixels. A full-size image is millions of numbers, and running a large network over all of them dozens of times would be slow. So a companion network squeezes each image into a much smaller summary first, roughly 48 times fewer numbers. That summary keeps the content and the composition and drops the pixel-level repetition. (The squeezing network is a VAE, short for variational autoencoder.)

The whole story of adding and removing noise plays out on that compressed canvas. (Its technical name is the latent.) Only at the very end does the VAE’s decoder expand the finished canvas back into real pixels. That takes a single pass, but a memory-hungry one, which is why careful phone apps decode the image in tiles. This compression is the main reason diffusion runs on consumer hardware at all.

Seeds, steps, and the other knobs

The knobs in an image app line up directly with the machinery above.

  • Seed. The starting noise is random, but computers fake randomness from a starting number. That number is the seed. Keep the model, prompt, settings, and seed the same, and you get a reproducible result when the full pipeline is deterministic. Change only the seed and you get a different take on the same idea. That repeatability is why apps let you keep or reroll seeds.
  • Steps. How many passes the denoising loop makes. More steps refine the image further, but the gains drop off sharply. Distilled models are trained to need only one to four.
  • Model. Which trained denoiser you run. Bigger models hold more visual knowledge and follow prompts more faithfully. Smaller and distilled ones trade some of that for speed and size. SDXL Turbo is the classic example of a model rebuilt for few-step generation.

None of these knobs add intelligence. They pick among the things one fixed network already knows how to do.

The same loop, running in your hand

Onhand’s Turbo and Detailed Images models use the diffusion process described here, through MLX on the device GPU. The library also contains other model families and Core ML packages with different schedules. Photo-to-photo editing uses an existing image as input; Refine starts from a finished result.

Image inference runs on your iPhone or iPad. Prompts and images are not sent to an inference server. Purchases, downloads, and limited analytics are separate network functions, explained in the privacy policy. The app keeps technical seed handling internal and offers Reuse settings, Make variations, and Refine as consumer workflows.

Cloud services generate on far larger hardware; Onhand vs Midjourney compares that approach with an on-device one on quality, privacy and cost. For what the on-device version needs before it works offline, see the offline AI image generator for iPhone.

Sources

Primary sources for the claims above.