Key takeaway: Video at 30 fps contains enormous redundancy. The engineering question is not how to process it all but which frames carry information.
The Arithmetic
One hour at 30 fps is 108,000 frames. Sending each to a vision model at typical per-image pricing produces a cost that makes the approach untenable for anything beyond a demonstration.
Most of those frames are near-duplicates. In a static scene, thousands of consecutive frames differ only by sensor noise. Processing them individually pays repeatedly for the same information.
Selection Strategies
| Strategy | Cost | Suitable for |
|---|---|---|
| Every frame | Prohibitive | Precise motion analysis only |
| Fixed interval (1 per second) | 1/30 | General content understanding |
| Scene change detection | Very low | Edited video, presentations |
| Motion-triggered | Low | Surveillance, monitoring |
| Keyframe extraction | Low | Summarisation, indexing |
Scene change detection is the highest-leverage technique for edited content. Comparing consecutive frame histograms or computing perceptual hash distance identifies cuts cheaply, and one representative frame per shot captures most of the semantic content of a video at a tiny fraction of the cost.
For continuous unedited footage — a fixed camera — motion detection serves the same role. An empty corridor generates no frames worth analysing, and the ninety-five percent of surveillance footage where nothing happens should cost nothing.
Fixed-interval sampling at one frame per second is a reasonable default when content type is unknown. It captures anything lasting longer than a second, which covers most events people care about.
What Frame Sampling Loses
Sampling discards temporal information, and some questions require it.
Distinguishing picking up from putting down needs frame order. Counting repetitions needs continuity. Recognising a gesture requires the trajectory rather than a snapshot. Reading a rapid action can miss it entirely between samples.
For these, a model with genuine temporal modelling — processing a clip rather than independent frames — is necessary. The practical approach is hierarchical: sample sparsely to locate candidate regions, then process those short segments densely.
Audio is the most undervalued signal in video analysis. Transcribing speech is far cheaper than analysing frames and frequently answers the question directly. For instructional content, meetings and presentations, the transcript carries most of the information and the frames add context.
A Workable Pipeline
Detect scene boundaries with a cheap classical method. Extract one representative frame per scene. Transcribe the audio track. Send frames and transcript together to a vision language model for understanding. Escalate to dense temporal processing only for segments where the question demands it.
That structure reduces cost by orders of magnitude relative to uniform frame processing while retaining the information most queries need.
The Bottom Line
Never process every frame. Use scene detection or motion triggers to select frames, transcribe audio as a parallel and cheaper signal, and reserve dense temporal models for the specific segments where frame order genuinely matters.
