Writing Prompts for On-Device Diffusion Models

Most prompt advice online was written for giant cloud models. Those systems carry tens of billions of internal dials (parameters). Often a language model rewrites your prompt before the image model sees it. This guide focuses on smaller Turbo models. They build pictures in one to four passes. An app may combine your text with a selected style or its own prompt template before encoding it. (The technical names: these are on-device diffusion models, and each pass is a denoising step.)

Small, few-pass models work best with prompts that are concrete, visual, and front-loaded, and they ignore a lot of what people habitually type. This guide covers the patterns that work on a phone, each with a before-and-after rewrite.

Describe the result, not the change

A model like this has no memory of your last image. It does not understand instructions, and it has no idea what “it” refers to. Every image starts fresh from noise plus your text. So prompts written as requests (“make it more dramatic,” “add a hat,” “remove the background”) ask for a change to something the model cannot see. The fix is to describe the finished picture as if it already exists.

• Instead of: “make the lighting more dramatic”. Write: “portrait of an old fisherman, dramatic low-key lighting, single hard light from the left, deep shadows”
• Instead of: “add a cat to the sofa”.Write: “a ginger cat curled up on a green velvet sofa, warm afternoon light, cozy living room”
• Instead of: “can you do this in a watercolor style please”. Write:“watercolor painting of a lighthouse on a cliff, loose brushwork, soft washes of blue and gray”

Notice that the rewrites also drop the chatty filler. Words like “please,” “can you,” and “I want” show nothing on screen, and on a small model they cost you. Your prompt gets chopped into chunks (tokens), and every chunk spent on politeness is a chunk not spent on the image.

Put the style in the positive prompt

On cloud services, many people learned to steer quality with a negative prompt. That is a second text box where you list what to avoid: “blurry, extra fingers, low quality.” On the fast on-device models (the turbo class), that habit does nothing. A negative prompt only works when the model guesses twice on every pass and compares the two guesses. (The technical name is classifier-free guidance.) Few-step turbo models skip that second guess to hit their speed. The details are in why negative prompts do nothing in turbo models. The practical rule: everything you want must be stated positively.

• Instead of: negative prompt “blurry, low quality”. Write: “sharp focus, detailed, high quality” in the main prompt.
• Instead of: negative prompt “cartoon, illustration”. Write:“photograph, 35mm, natural skin texture” in the main prompt.

Naming the style you want beats listing the styles you fear. That holds even on models where negative prompts do work. On a turbo model it is the only option that works at all.

Concrete nouns beat abstractions

Small models have seen fewer examples of everything. So they lean harder on words that turned up with a strong, consistent look across the captions they learned from. Vague words (“beautiful,” “epic,” “stunning,” “vibes”) point at nothing in particular and mostly add clutter. Concrete nouns and named materials point at specific pixels.

• Weak: “a beautiful magical forest, amazing atmosphere”
• Strong: “pine forest at dawn, fog between the trunks, shafts of golden light, moss-covered rocks”

The strong version wins because a camera could point at every phrase in it. Fog, trunks, light shafts, moss: each noun gives the model a target. Before you tap generate, try a quick test. Ask of each word, “could I circle this in the finished image?” If not, consider replacing it.

Borrow the vocabulary of photographers and painters

Framing and lighting terms are the exception. They are abstract, but they earn their place, because they appeared in millions of image captions during training. Here is a short list that works on small models: close-up, wide shot, low angle, overhead view, centered composition, shallow depth of field for framing; golden hour, overcast, backlit, rim light, soft window light, neon, candlelight for lighting; and oil painting, watercolor, ink sketch, 35mm photo, isometric illustration, pixel art for medium. One or two terms from each group is usually enough. Stack ten style tags and they mostly fight each other.

Mind the length limit

There is a hard ceiling most prompt guides never mention. Before any painting starts, a separate piece of software reads your words and turns them into numbers. (The technical name is a CLIP-style text encoder.) In Stable Diffusion–family models it has room for only 77 tokens, or roughly 50 to 60 English words. Anything past that is cut off before the model sees it. So the epic 200-word prompts that circulate online are mostly decorative. Their second half was never read.

That cutoff also means order matters. Put the subject first, then the setting, then style and lighting. If something must survive, it belongs in the first line. A good on-device prompt reads like a dense caption: “ink sketch of a narrow Tokyo alley at night, rain, glowing signs, one figure with an umbrella, high contrast.” It is a compact description; its token count depends on the tokenizer.

Iterate with seeds, not adjectives

Every image starts from a field of random static. One number picks that static, and it is called the seed. A fixed seed can help compare prompts in tools that expose it; reproduction still depends on the complete pipeline. Same prompt, different seed: a different composition. That gives you an orderly way to improve results.

1. Explore: keep the prompt fixed and generate several images on random seeds. At this stage you are choosing a composition.
2. In tools with seed locking: when a composition gets close, hold that seed still and change one phrase at a time. Swap the lighting term. Tighten the subject. The static stays identical, so any difference you see comes from the words you changed.

Onhand has no seed-lock control. Use Make variations for fresh results and Refine to work from a saved image, and judge several attempts before deciding that a prompt needs more adjectives.

Prompting in Onhand

Onhand’s model library includes Turbo, Detailed Images, Detailed Scenes, and Core ML style packs. The advice about short prompts and guidance-free sampling is most directly applicable to the Turbo models. Other choices use different text encoders and internal settings.

Onhand combines your text with the selected style and model-specific prompt templates. Use Make variations to explore, Reuse settings to restore a recipe, and Refine to change a finished image. The prompt examples in this article are suggestions to try, not a record of generated results.

Large cloud models respond to prompts differently; Onhand vs Midjourney covers where a cloud service still has the edge on prompt handling.