Key takeaway: “Similar” is not a property of images. It is a property of the embedding space, and choosing the wrong space produces results that are technically correct and practically useless.
Four Incompatible Notions of Similar
Given a photograph of a red leather armchair in a bright room, a search system might reasonably return:
- Other armchairs, any colour — object similarity
- Other red leather items — attribute similarity
- Other bright minimal interiors — style similarity
- Other photographs of that exact chair — instance similarity
All four are defensible. Which one you get is determined by the model, not by the query.
| Model type | Optimises for | Good for |
|---|---|---|
| CLIP-style contrastive | Text-image alignment | Text search, semantic categories |
| ImageNet classifier features | Object identity | Category matching |
| Self-supervised (DINO-style) | Visual structure | Instance and near-duplicate |
| Face recognition | Identity under variation | People |
| Perceptual hash | Near-exact pixels | Duplicate detection |
Using CLIP embeddings to find duplicates returns semantically related but visually distinct images. Using a perceptual hash to find “similar products” returns nothing unless the images are near copies. Both are the model working correctly against the wrong objective.
What CLIP Actually Gives You
CLIP-family models place images and text in a shared space, which enables searching a photo library with a phrase. That capability is genuinely useful and widely deployed.
Its weakness is precision on fine distinctions. CLIP knows chair robustly. It is much weaker on this specific chair model versus that one, on counting, on spatial relationships, and on text rendered within the image. It was trained to associate images with captions, and captions rarely specify those things.
For product catalogues where variant-level accuracy matters, a model fine-tuned on your own catalogue with your own notion of “same product” outperforms any general model, because you are defining the similarity metric rather than inheriting one.
Practical Construction
Normalise before comparing. L2-normalise embeddings and use cosine similarity. Mixing normalised and unnormalised vectors produces silently wrong rankings.
Combine signals rather than choosing one. Retrieve broadly with a semantic embedding, then re-rank with structured attributes — colour histogram, category, price band, availability. Pure visual similarity ignores everything the business knows about the item.
Filter before searching where possible. Restricting to in-stock items in the right category before nearest-neighbour search improves both relevance and latency.
Version the embedding model. Changing models invalidates the entire index. Store the model identifier with every vector so you can migrate incrementally rather than rebuilding blindly.
Evaluating Honestly
Build a small labelled set of query images with human-judged correct results, then measure precision at 10. Eyeballing results is unreliable because plausible-looking neighbours mask systematic problems — you will not notice that colour is being ignored until you measure it.
The Bottom Line
Define what similar means for your application before selecting a model, then pick the embedding family that optimises for that definition. Re-rank with structured attributes, and measure precision against labelled examples rather than trusting appearances.
