Key takeaway: Interactive latency is expensive because it requires idle capacity. Most production inference is not interactive, and processing it as batch cuts cost sharply for no quality loss.
Why Batch Costs Less
Synchronous serving must answer immediately, which means holding capacity ready for peak load. Between requests, expensive accelerators sit idle, and that idle time is priced into every request.
Batch processing removes the deadline. Work is queued and executed when capacity is available, which lets the operator fill troughs, pack large batches for better throughput, and run at high utilisation. Providers pass a share of that back as a discount, commonly around half.
Local serving benefits identically. Batch size is the dominant factor in GPU throughput, because per-token compute is memory-bandwidth-bound and larger batches amortise weight loading across more sequences.
Identifying Eligible Work
The test is whether a human is waiting. Surprisingly often, nobody is.
| Workload | Batchable | Note |
|---|---|---|
| Nightly document classification | Yes | Obvious candidate |
| Embedding a corpus | Yes | Never interactive |
| Report and summary generation | Yes | Scheduled anyway |
| Evaluation and benchmark runs | Yes | Frequently overlooked |
| Enriching newly ingested records | Usually | Minutes are acceptable |
| Content moderation queue | Often | Depends on policy |
| Chat response | No | Human waiting |
| Inline autocomplete | No | Sub-second required |
Evaluation runs deserve specific mention. Teams run large offline evaluations through synchronous endpoints out of habit, paying interactive prices for work that has no deadline at all.
The ingestion case is where the largest savings usually hide. Enriching a record within minutes rather than milliseconds is almost always acceptable to the business, and nobody asked because the synchronous path was the default.
Practical Considerations
Batch jobs need durable state. A job submitted with results arriving later requires request identifiers, status tracking and idempotent result handling so that a retry does not duplicate work.
Partial failure must be handled per item rather than per job. A batch of ten thousand where forty items fail should not fail the batch, and those forty need a defined retry path.
Deadlines still exist even without latency requirements. A nightly job must finish before morning, so completion-time monitoring matters even though per-request latency does not.
The Hybrid Pattern
The strongest arrangement serves the first view synchronously and enriches asynchronously. A user uploading a document gets immediate basic feedback from a cheap fast path, while thorough analysis runs as batch and appears shortly afterwards.
This gives the perceived responsiveness of synchronous processing for the majority of cost that batch pricing covers, and it usually matches what users actually need rather than what the architecture assumed.
The Bottom Line
Audit your inference traffic for requests where nobody is waiting, and move them to batch. Start with evaluation runs and corpus embedding, then examine ingestion enrichment — the savings are large and the engineering change is modest.
