Batching
-
Your GPU Is Idle: Why Inference Throughput Is Memory-Bound
Language model inference rarely saturates GPU compute. It saturates memory bandwidth, and understanding that changes every optimisation decision you make…
Read More » -
Dynamic Batching Is Why Two Identical Requests Get Different Latency
A request arriving alone is served immediately. The same request arriving during a full batch window waits for the batch…
Read More »