Table of Contents
- The Gap Between Generating and Shipping
- Latency Makes Synchronous Requests Impossible
- Why Users Cannot Write Prompts
- Moderation Runs in Both Directions
- The Economics of GPU Inference
- Provenance and Disclosure
- Handling Unacceptable Output
- A Reference Pipeline
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: The model is the easy part. Production image generation is an asynchronous job system with bidirectional moderation, prompt transformation, cost controls, and a plan for the substantial fraction of outputs that are technically fine and practically unusable.
The Gap Between Generating and Shipping
Calling an image generation API takes four lines of code. Putting that capability in front of users is a different engineering problem entirely, and the difference catches teams out consistently.
The demo works because a knowledgeable person writes a careful prompt, waits patiently, and discards the attempts that did not work. Every one of those behaviours breaks in production. Real users write two-word prompts, expect immediate results, and treat the first output as the product’s quality level.
What sits between the API and a shippable feature: an asynchronous job system with progress feedback, prompt transformation because user input is inadequate, moderation on both input and output, cost controls and abuse prevention, provenance metadata, storage and delivery infrastructure, and a workflow for the outputs users will reject.
None of this is exotic. All of it takes longer than the model integration.
Latency Makes Synchronous Requests Impossible
Image generation takes seconds to tens of seconds. That single fact determines your architecture.
A synchronous HTTP request holding a connection for 30 seconds fails in several ways: proxies and load balancers time out, mobile networks drop connections, a user navigating away wastes the compute entirely, and concurrency is limited by held connections rather than by GPU capacity.
The required shape is a job queue.
POST /generate → validate, enqueue, return job_id immediately
GET /jobs/{job_id} → status: queued | running | complete | failed
or push updates over a websocketDetails that matter in practice:
Report queue position, not just status. “Position 4 in queue, roughly 40 seconds” is tolerable. A spinner with no information is not, and abandonment rises sharply without it.
Show progressive previews where the model supports it. Diffusion models can emit intermediate steps. A blurry preview at three seconds transforms perceived responsiveness even though total time is unchanged.
Set a hard timeout and fail explicitly. Jobs that hang consume GPU capacity indefinitely. Fail at a threshold, refund whatever quota was consumed, and tell the user.
Make jobs idempotent and resumable. Workers crash. A job that cannot be retried safely becomes a lost request and a support ticket.
Separate queues by priority. Paying users and free users on one queue means free traffic degrades paid experience. Separate queues with separate capacity allocations prevent that.
Why Users Cannot Write Prompts
Prompt quality determines output quality, and users write poor prompts. This is not a criticism — they have no reason to know what a model responds to.
A user types “dog.” An effective prompt for the same intent might specify subject, composition, lighting, style, and quality descriptors. The gap between these produces a generic, disappointing result that the user attributes to the product being bad.
Approaches that close the gap, in increasing sophistication:
Template augmentation. Wrap user input with quality and style modifiers appropriate to your product. Simple, cheap, and delivers most of the improvement available. This is the highest-return single change.
Language model rewriting. Pass user input through a model that expands it into a detailed prompt. More effective and adds latency and cost. Worth showing the rewritten prompt to the user, since they will otherwise be confused about why they got what they got.
Structured input. Rather than a free text field, offer controls — subject, style, mood, aspect ratio. This constrains the space toward outputs your product handles well and eliminates most prompt-quality variance.
Negative prompts. Maintaining a standing list of undesirable characteristics improves consistency substantially and requires no user involvement.
The general principle is that exposing a raw prompt field to users transfers a skilled task to people without the skill. Most successful products constrain input rather than accepting anything.
Moderation Runs in Both Directions
Input filtering alone is insufficient, and understanding why matters.
Input moderation blocks prompts requesting content you will not produce. Necessary and easily circumvented, because natural language offers many routes to the same output. Blocklists catch the obvious attempts and fail against paraphrase.
Output moderation examines the generated image. This is the layer that actually protects, because it inspects what was produced rather than what was asked. A benign-seeming prompt can produce problematic output, and only output inspection catches that.
Both layers are needed and they serve different functions. Input filtering deters and documents intent; output filtering enforces.
Categories requiring attention: sexual content, violence, recognisable public figures, trademarked and copyrighted characters, and content that could constitute harassment of a specific person. The last is frequently overlooked and is where products cause the most concrete harm to identifiable individuals.
Two operational points. Output moderation adds latency to an already slow pipeline, so run it in parallel with upload rather than serially where possible. And moderation classifiers produce false positives that block legitimate requests, which means an appeal or review path is necessary unless you accept silently frustrating users.
The Economics of GPU Inference
Image generation is compute-intensive in a way that makes cost a first-order design constraint rather than an operational detail.
| Approach | Cost characteristic | Best suited to |
|---|---|---|
| Hosted API per image | Highest per image, zero fixed | Low or unpredictable volume |
| Serverless GPU | Moderate, cold start penalty | Bursty traffic |
| Reserved GPU instances | Lowest per image at high use | Steady high volume |
| Self-hosted hardware | Lowest marginal, high capital | Very large steady volume |
The crossover between hosted APIs and reserved capacity arrives faster than most teams expect, because a continuously busy GPU produces a large number of images relative to per-image API pricing. The catch is utilisation: reserved capacity idling overnight wastes the entire advantage, which is why traffic pattern matters more than volume.
Levers that reduce cost regardless of hosting choice:
Reduce inference steps. Quality gains diminish substantially past a point. Many pipelines run more steps than the output quality justifies.
Generate at lower resolution, then upscale. Upscaling is far cheaper than generating at the target resolution directly, and the quality difference is frequently imperceptible.
Batch requests. GPUs process batches far more efficiently than sequential single requests. Batching adds latency and multiplies throughput.
Cache aggressively. Identical prompts recur more than you would expect, particularly from template-augmented input where users choose from limited options.
Enforce quotas per user. Without them, a single automated client can consume your entire capacity budget. This is an abuse-prevention measure as much as a cost one.
Provenance and Disclosure
Attaching metadata declaring an image as AI-generated is moving from good practice toward expectation and, in some jurisdictions, requirement.
Two mechanisms, with different properties. Embedded metadata following content credential standards records generation details in the file, is cryptographically signable, and is straightforward to strip. Invisible watermarking encodes a signal in the pixels that survives resizing and re-encoding, is harder to remove, and requires a detector to read.
Neither is robust against a determined adversary. Both are worth implementing, because the realistic threat model is not an adversary — it is downstream confusion about whether an image was generated, which metadata resolves for the majority of ordinary cases.
Beyond the technical mechanism, product-level disclosure matters: a visible indicator in the interface, retained generation records so you can answer questions about a specific image later, and clear terms about what users may do with output. That last point is genuinely unsettled ground legally, and the honest position is to tell users so rather than implying certainty that does not exist.
Handling Unacceptable Output
A substantial fraction of generated images are not usable — anatomically wrong, compositionally broken, or simply not what the user wanted. This is normal model behaviour and your product needs a workflow for it rather than treating it as an edge case.
Generate several candidates. Four images presented as options is a better experience than one image presented as the answer, because it converts a quality problem into a choice. This is the single most effective pattern available.
Make regeneration prominent and cheap. Users will retry. A retry that requires re-entering the prompt loses them; a retry button that varies the seed keeps them.
Offer refinement rather than restart. Variations on a selected image, inpainting to fix a specific region, or adjusting one attribute while holding others. Restarting from scratch discards the progress the user made.
Track rejection rate as a primary metric. The proportion of generated images users discard is your real quality signal, and it is available without any labelling effort.
Never charge quota for obvious failures. If output fails moderation or errors out, that should not consume the user’s allowance. Charging for failures generates disproportionate frustration.
A Reference Pipeline
1 Receive request validate, check quota, check rate limit
2 Moderate input prompt classification, blocklist
3 Transform prompt template or model rewrite, negative prompt
4 Enqueue priority by tier, return job_id
5 Generate N candidates, batched, on GPU worker
6 Moderate output image classification per candidate
7 Post-process upscale, format conversion, compression
8 Attach provenance content credentials, watermark
9 Store and serve object storage, CDN, signed URLs
10 Record prompt, params, outcome, user actionStep 10 is the one most likely to be omitted and the most valuable over time. Recording the prompt, the parameters, whether moderation triggered, and what the user did with the result gives you the data to improve prompt transformation, tune moderation thresholds, and measure quality — none of which is recoverable retroactively.
Common Pitfalls
Synchronous request handling. Guarantees timeouts and wasted compute. A queue is not optional at this latency.
Input moderation only. Prompts can be paraphrased. Output inspection is what actually enforces policy.
Exposing raw prompts with no augmentation. Transfers a skilled task to users who lack the skill, and they blame the product.
Single output per request. Presenting one image as the answer sets an accuracy expectation the model cannot meet.
No per-user quotas. One automated client can consume your entire budget.
Generating at full resolution directly. Lower resolution plus upscaling is substantially cheaper for comparable quality.
No generation records. Makes quality improvement and incident investigation impossible.
Conclusion
Image generation in a product is an asynchronous job system that happens to call a model. The queue, the prompt transformation, the bidirectional moderation, the cost controls, and the refinement workflow are where the engineering effort actually goes.
Prioritise in this order: get the queue and progress feedback right, because latency is unavoidable and abandonment is not. Augment prompts, because users will not write good ones. Moderate output as well as input, because that is the layer that works. Generate multiple candidates, because choice converts a quality problem into a selection task. And record everything, because rejection rate is the quality metric you cannot reconstruct later.
The model will improve without your involvement. Everything listed above will not.
Frequently Asked Questions
Hosted API or self-hosted models? Start with a hosted API — it removes GPU operations entirely while you learn your traffic pattern. Move to reserved or self-hosted capacity when volume is steady and high enough that per-image pricing dominates your costs.
How many candidates should be generated per request? Four is a common sweet spot: enough choice to usually contain something acceptable, without quadrupling cost or overwhelming the interface. Two is defensible for expensive high-resolution generation.
Is output moderation genuinely necessary if input is filtered? Yes. Benign prompts can produce problematic images, and prompts can be phrased to evade input filters. Output inspection examines what was actually produced, which is what your policy concerns.
How is the copyright position on generated images? Genuinely unsettled and varying by jurisdiction, particularly regarding authorship and training data. The practical advice is to state clearly in your terms what you do and do not warrant, and to avoid implying legal certainty.
Should prompt rewriting be shown to the user? Generally yes. Users who see the expanded prompt understand why they got a particular result and can adjust. Hidden transformation produces confusion when output does not match the input they typed.
How should generation failures be handled billing-wise? Do not charge for them. Moderation blocks, timeouts, and errors should not consume quota. The frustration from being charged for a failure substantially exceeds the cost of absorbing it.
What latency should be targeted? Under 10 seconds feels responsive with progress feedback; up to 30 is tolerable with queue position shown. Beyond that, treat it as a background task with notification on completion rather than a foreground wait.
