Key takeaway: Edge deployment is constrained by memory bandwidth, thermal limits and available operators rather than by raw compute. Validate on the target device early, because late discovery forces a redesign.
The Constraints That Actually Bind
Teams optimise for parameter count and floating-point operations, then find the model too slow for reasons neither number predicted.
Memory bandwidth is usually the real limit. Moving weights and activations between memory and the accelerator dominates the time budget, so a model with fewer parameters but poor memory access patterns can be slower than a larger one that streams well.
Thermal behaviour is the second. An embedded accelerator hits its power limit within seconds of continuous inference and then throttles. A benchmark measuring one batch reports a number the device cannot sustain for a minute, let alone continuously.
Operator support is the third and the most disruptive. Edge runtimes implement a subset of operations. A model using one unsupported layer either fails to convert or falls back to CPU for that layer, which can cost more than the rest of the network combined.
| Constraint | How it surprises people |
|---|---|
| Memory bandwidth | Small model still slow |
| Thermal throttling | Fast for 10s, then halves |
| Unsupported operator | Silent CPU fallback |
| Memory ceiling | Works alone, fails alongside the app |
| Quantisation loss | Accuracy drops on rare classes |
Reducing Size Without Losing What Matters
Quantisation converts weights from 32-bit floats to 8-bit integers, cutting size by four and usually improving speed more than that because of the bandwidth effect.
Post-training quantisation requires only a small calibration set and is quick to try. Quantisation-aware training simulates the reduced precision during training and recovers most of the accuracy loss, at the cost of retraining.
The important caveat is that quantisation loss is not distributed evenly. Overall accuracy may fall a fraction of a percent while accuracy on a rare but important class falls substantially. Evaluating only the aggregate figure hides that, and the rare class is often the one the application cares about.
Pruning removes weights that contribute little. Unstructured pruning produces sparse tensors that most edge accelerators cannot exploit, so the size reduction does not become a speed reduction. Structured pruning — removing whole channels — yields a smaller dense model that runs faster on real hardware.
Knowledge distillation trains a small model to match a large one’s outputs and frequently beats training the small architecture directly, because the teacher’s soft predictions carry more information than hard labels.
Getting the Process Right
Measure on the target device from the first week rather than at the end. A model architecture chosen without device measurements is a guess, and discovering the mismatch after training is complete means starting again.
Benchmark sustained throughput over several minutes, not a single batch. The steady-state number is what the application experiences.
Verify the converted model’s outputs against the original on a held-out set. Conversion bugs are real and produce silently degraded models rather than errors.
Version the deployed model and support remote update, because a fleet that cannot be updated is a fleet stuck with whatever accuracy it shipped with.
The Bottom Line
Profile on target hardware before choosing an architecture, measure sustained rather than peak throughput, prefer structured pruning and quantisation-aware training, and check per-class accuracy after quantisation rather than trusting the aggregate.
