Key takeaway: Fine-tuning changes how a model responds. Retrieval changes what it knows. Using the wrong one produces a model that sounds authoritative and gets the details wrong.
The Common Misdiagnosis
A team wants a model that answers questions about their product. They assemble documentation into training examples and fine-tune.
The result speaks convincingly in their domain vocabulary and gets specifics wrong. Version numbers drift. Configuration flags are invented. Deprecated behaviour is described as current.
The reason is that fine-tuning adjusts weights toward patterns in the training data. It teaches style, format and task structure effectively. It stores facts poorly and diffusely, with no mechanism for citation and no way to update a single fact without retraining.
What Each Approach Is Good At
| Requirement | Fine-tuning | RAG |
|---|---|---|
| Consistent output format | Strong | Weak |
| Domain tone and vocabulary | Strong | Weak |
| Factual accuracy on specifics | Weak | Strong |
| Citing sources | Not possible | Native |
| Updating a single fact | Full retrain | Edit one document |
| Reducing prompt length | Strong | Increases it |
| Latency | Lower | Higher |
| Setup cost | High | Moderate |
The update characteristic decides most real cases. Product documentation changes weekly. A retrieval corpus absorbs that with a document edit. A fine-tuned model requires a new training run, evaluation, and deployment for every factual correction.
Where Fine-Tuning Genuinely Wins
Rigid output structure. When every response must match a specific schema or format, fine-tuning achieves reliability that prompting struggles to match, and it removes the format instructions from every request.
Specialised classification. A model fine-tuned on tens of thousands of labelled examples of your particular taxonomy will outperform a prompted frontier model, at a fraction of the cost per call.
Style and voice. Matching a brand voice or a specific writing register is a behavioural property, which is exactly what fine-tuning encodes.
Prompt compression. If a 3,000-token system prompt is required on every call, fine-tuning that behaviour into the weights eliminates a recurring cost.
Latency and cost at volume. A small fine-tuned model matching a large prompted model on a narrow task is dramatically cheaper and faster.
Using Both
The strongest production systems combine them. Fine-tune a smaller model to follow your response format, cite retrieved sources correctly and adopt the right register. Retrieve current facts at request time. The fine-tuning handles behaviour; retrieval handles truth.
LoRA and similar adapter methods make this practical — training a small set of additional parameters rather than the full model, which reduces cost enough that iteration becomes realistic.
Ordering the Work
Start with prompting and measure. Add retrieval when the gaps are factual. Consider fine-tuning only when the remaining gaps are behavioural and you have several thousand quality examples.
Reversing that order is common and expensive, because fine-tuning is the option with the highest setup cost, the slowest iteration loop and the least diagnostic transparency.
The Bottom Line
Diagnose whether your failures are about knowledge or about behaviour. Knowledge failures need retrieval. Behaviour failures need better prompting first and fine-tuning second, once you have the data to justify it.
