Key takeaway: Synthetic data can supply volume and cannot supply information the generator does not have. Using it for exactly the rare cases you lack is the situation where it helps least.
The Circularity Problem
A classifier performs poorly on an uncommon category with forty real examples. The obvious fix is generating a thousand synthetic ones.
The generator produces samples drawn from its own understanding of that category, which is derived from a similar limited distribution. The thousand new examples are variations on the forty, concentrated in the region the generator already models well and absent from the region where the real gap lies.
Training on them raises confidence without raising accuracy on genuinely novel instances. The model becomes more certain about a narrower slice, which is the opposite of what was needed.
Where Synthetic Data Genuinely Helps
The technique is not without value; it is specific about where value exists.
| Use | Effectiveness |
|---|---|
| Geometric and photometric augmentation | High |
| Rendering from 3D assets with known labels | High |
| Rare event simulation with a physics model | High |
| Balancing class counts for a common class | Moderate |
| Generating novel rare cases from a model | Low |
| Privacy-preserving substitute for real data | Depends heavily |
The distinction is whether the synthetic process introduces information from outside the model. A physics simulation of a mechanical failure encodes real domain knowledge, so it can produce examples the training set never contained. A rendered scene from 3D assets provides exact labels and viewpoints that would be expensive to capture.
Generation from a learned model introduces nothing external. It can smooth and interpolate, which is useful for regularisation, and it cannot extrapolate to genuinely unseen conditions.
The Degradation Risk
Training successive model generations on the previous generation’s output causes measurable quality loss. Diversity narrows, tail behaviour disappears, and the distribution contracts toward the mean with each cycle.
This matters practically because it can happen unintentionally. Data scraped from public sources increasingly contains generated content, so a corpus assembled today may include model output without anyone labelling it as such. Recording provenance is the only defence, and it needs to happen at collection time.
Mixing a small proportion of synthetic data with a majority of real data appears to avoid the worst of this. Ratios above roughly a third synthetic tend to show degradation on held-out real evaluation, though the threshold varies by task.
Evaluating Honestly
The essential discipline is that evaluation must use real data exclusively. A model trained and validated on synthetic examples will score well on synthetic examples and tell you nothing about production performance.
Keep a real held-out set that never contains generated content, and treat any improvement measured only against synthetic evaluation as unverified.
Where synthetic data is used to address a specific weakness, measure that weakness on real examples before and after. If accuracy on real rare cases did not improve, the synthetic data added confidence rather than capability — and that is a worse outcome than doing nothing, because it hides the gap.
The Bottom Line
Use synthetic data where the generation process encodes real knowledge — simulation, rendering, principled augmentation — and be sceptical where it comes from a model trained on the same limited data. Keep synthetic below roughly a third of the mix, record provenance, and evaluate exclusively on real held-out examples.
