RAG or fine-tuning?

These get compared constantly and they are not alternatives. They solve different problems, and the comparison persists because both are the answer to "the model doesn't know about our stuff", which is two different complaints wearing one sentence.

RAG changes what the model knows. Fine-tuning changes how it behaves. If you cannot say which one you need, you need neither yet.

The distinction that resolves most cases

Retrieval is for facts. Things that are true today, might change tomorrow, and need to be cited. Your documentation, your policies, your product catalogue, your customer's account history.

Fine-tuning is for form. Consistent output structure, a house voice, a classification boundary the base model keeps getting wrong, or a specialised task where you have thousands of correct examples.

A quick test: if the right answer changes when someone edits a document, you need retrieval. If the right answer depends on knowing your conventions rather than your facts, you might need fine-tuning.

Why RAG is almost always the right first move

• Updates are instant. Change the document, the answer changes. Fine-tuning requires a retraining cycle for every fact that moves.
• It can cite. Provenance is available essentially for free — you know which passage produced the answer. With a fine-tuned model, the knowledge is smeared across weights and cannot be pointed at, which matters enormously the first time a customer or regulator asks why.
• Access control works. You can filter retrieval by who is asking. A fine-tuned model knows everything it was trained on, for everyone, permanently. This alone disqualifies fine-tuning for a great deal of enterprise work.
• It is far cheaper to try and to abandon.

When fine-tuning genuinely earns its place

Four cases where we have seen it be right:

1. Rigid output format at scale. You need the same JSON shape every time, and prompting gets you to 97% while you need 99.9%.
2. A specialised classification where you have thousands of labelled examples and the base model's boundary is subtly wrong for your domain.
3. Voice and register, when it must be consistent across enormous volume and prompting is inconsistent.
4. Latency and cost at scale — a small fine-tuned model matching a large prompted one, where volume makes the difference material. This is a real and underrated case, but it is an optimisation, not a starting point.

Notice that none of those are "the model needs to know our data".

The option nobody sells you

Most projects that arrive asking this question need neither. They need:

• Better retrieval — hybrid search rather than pure vector.
• A clearer prompt with two worked examples in it.
• A schema that fails loudly instead of a free-text blob.
• An eval set, so the question can be settled by measurement rather than argument.

That last one is the real answer. This debate is usually a proxy for not being able to measure anything: with a golden set you can test both in a week and stop arguing. Without one, you are choosing between two expensive commitments on instinct.

If you must decide today

• Facts that change → RAG.
• Behaviour that will not stabilise under prompting, and thousands of examples to prove it → fine-tune, after RAG.
• Neither is clearly true → build the eval set first. It is a week, and it answers the question properly.

Related: how to evaluate an LLM pipeline · why RAG returns wrong answers