How On-Device AI Image Generation Works on iPhone

On-device AI image generation runs the model on your iPhone itself, so your prompt, your photos, and your results do not have to reach a server. Most AI image apps work the other way. Type a prompt and your words leave your phone right away. They travel to a data center, where a rack of graphics chips (server GPUs) turns them into an image, and the picture is sent back to you.

On-device generation turns that around. The model lives on your iPhone instead. A model is the trained program that makes the pictures (a neural network). It carries with it all the numbers it picked up during training (its parameters). You download it once and it stays there. When you tap generate, your own phone does the work. Nothing about your prompt, your photos, or your results has to reach a server. This article walks through what that takes, and why it has only recently become possible.

What a diffusion model does

Most image generators today work by cleaning up static. In training, the model is shown billions of pictures, each one buried under random speckle a little at a time. Its only job is to look at a messy picture and point out which parts are the mess. (The technical names are diffusion models, and the mess is called noise.) Then you run that job in a loop. Start with nothing but static, give the model a written description to aim at, and it wipes the static away bit by bit. What is left behind is a picture that matches your words.

Each pass through the loop is one round of cleanup. The name for it is a denoising step. Older diffusion models needed 20 to 50 rounds to reach a clean image. Newer turbo models are trained to reach the same finish in far fewer rounds (distilled specifically for speed), and they make usable images in one to four rounds. On a phone, every round costs seconds and battery.

Why this used to require a data center

Three problems kept diffusion models locked in the cloud for years:

• Size. One image generator is several programs working together. A reader turns your prompt into numbers (the text encoder). A main engine does the cleanup work (the diffusion backbone). A converter turns the result into visible pixels (the decoder, or VAE). Stored at full precision, they add up to many gigabytes.
• Memory. Storing the model is only half the problem. It also has to fit in working memory while it runs. So do all the half-finished numbers it produces along the way (the activations). Server GPUs carry 40–80 GB of memory. iPhones have 6 to 12 GB, shared with the operating system and every other app.
• Compute. Each round of cleanup means billions of tiny multiplications and additions (multiply-accumulate operations). Until recently, phone processors could not get through that in a tolerable time.

All three problems eased at once. Shrinking the model’s numbers down to lower precision (quantization) cuts its weights to a quarter of their original size, with little quality loss. Turbo distillation cut the number of rounds by a factor of ten. And Apple silicon shares one fast pool of memory between the graphics chip and the main processor (unified memory). A compressed model can now be loaded and run entirely on the phone’s own chip.

What happens when you tap Generate

In an on-device app like Onhand, one image is made roughly like this. Every step runs on the device; the selected engine determines which processors do the work:

1. The text encoder turns your prompt into a grid of numbers the model can work with. This takes a fraction of a second.
2. The app lays out a canvas of pure static to work on. Its name is the “latent.” The static starts from a chosen number called the seed. Reproduction also depends on the model, settings, software, and numerical behavior; the seed alone is not enough.
3. The diffusion backbone runs its cleanup rounds. With a turbo model that is one to four passes. Almost all the time and energy goes here.
4. The decoder turns the finished canvas into pixels. Careful apps do this one tile at a time. Converting a whole image in one go is one of the biggest memory spikes in the entire process.

A full cycle includes loading when needed, prompt encoding, denoising, and decoding. Its duration varies by model, quality, device, and temperature. A useful speed comparison has to hold those conditions constant.

What you get in exchange

Local image processing. Running inference on the device removes the need to upload prompts and photos for that work. Check other data flows separately: analytics, purchases, backups, and sharing can use network services even when generation is local.

Offline capability. Once a model package is installed, it works in airplane mode, on the subway, or anywhere else. Onhand also uses network services for purchases and limited analytics.

No per-image economics. Cloud generators meter your usage because every image costs them time on their own hardware. When the work happens on your phone, there is no per-image server fee; electricity and device resources still cost something.

Determinism and control. The model on your device is yours. It will not change under you overnight, get retired, or hand your request to a different server when the service is busy.

The trade-offs

The biggest models in the world hold tens of billions of learned numbers, and none of them fit on a phone today. A model running on your phone will not follow a prompt as closely as those do. It will also struggle to spell out long stretches of text inside an image. Downloads are measured in gigabytes. Making pictures pushes the phone hard, so a long session warms the device and uses battery.

Phone models today sit about where cloud models were a couple of years ago. They are good and improving fast, and already more than enough for concept art, styled portraits, quick illustrations, and private photo edits. For the pictures you would rather not upload anywhere, they are the only option.

How Onhand applies all of this

Onhand downloads prepared models for compatible iPhones and iPads. Turbo, Detailed Images, and Detailed Scenes use MLX; Core ML packages offer other looks. Prompts and source images are processed locally. Results stay in the local gallery unless you save or share them. Purchases, model delivery, and limited analytics use separate network services described in the privacy policy.

To see what running locally means with no connection, the offline AI image generator for iPhone page walks through it. Onhand vs Draw Things compares Onhand with a long-established on-device image app for Apple devices.