RAG or fine-tuning? Almost always RAG

This question comes up on nearly every first call, usually phrased as “should we train it on our data?”

The short answer: probably not, and the thing you actually want is retrieval.

The distinction in one line

Retrieval (RAG) gives the model your facts at the moment it answers. You keep your documents in a searchable index; the relevant pieces get pulled in and put in front of the model with the question.

Fine-tuning changes the model’s behaviour by training it further on examples. It teaches format, tone, and how to approach a task.

So: knowledge is retrieval. Behaviour is fine-tuning.

Why most business problems are retrieval problems

When someone says “train it on our data”, they almost always mean “it should know our products, our policies, our history”. That is knowledge. Retrieval.

Retrieval also wins on the practical things:

• Your data changes. A price list updates weekly. With retrieval you update the index. With fine-tuning you retrain, and in the meantime the model is confidently quoting last month’s prices.
• You can cite sources. Retrieval lets you show which document an answer came from, which matters enormously for trust and is impossible with fine-tuning.
• You can control access. Retrieval can filter by permission before anything reaches the model. A fine-tuned model has absorbed everything it saw, and there is no putting it back.
• It is far cheaper to change. Wrong answer? Fix the source document.

That third point deserves emphasis. If you fine-tune on data that includes records some users should not see, you have baked that into the weights. There is no per-user filtering after the fact.

When fine-tuning is genuinely right

It is not never. Fine-tuning earns its place when:

• You need a very specific output format, consistently, at high volume, and prompting keeps drifting.
• You need a particular voice or house style that is easier to demonstrate with two hundred examples than to describe in a prompt.
• You have a narrow classification task where a smaller fine-tuned model is cheaper and faster than a large general one. This is a real and underrated win at volume.
• You need to cut latency or cost by getting a small model to do a job that currently needs a big one.

Notice none of those are about knowing things.

The pattern that usually wins

For most of what we build:

1. Retrieval over the client’s own documents, catalogue or records.
2. A carefully structured prompt carrying the task and the rules.
3. Deterministic code doing anything that touches data.
4. An evaluation set of real cases so changes can be measured.
5. Fine-tuning considered later, only if a specific measured problem calls for it.

We have almost never needed step five. When something was not working well enough, the fix was better retrieval or a clearer task definition, not a different model.

The cheaper thing that beats both, sometimes

There is a third option people forget: few-shot examples pulled from the client’s own history.

On an estimating system we built, every correction an estimator made was stored. On later runs, the most similar past corrections are retrieved and put in front of the model as examples. The system converges on that specific estimator’s judgement without any training run at all.

That is retrieval doing the job people imagine fine-tuning is for, and it improves the moment a human corrects something rather than at the next retrain.

How to decide, quickly

Ask: if I hired a new person, would this be in the handbook or in the training?

Handbook, look-it-up, changes over time → retrieval.

Training, how-we-do-things, learned by repetition → possibly fine-tuning.

Most of what businesses want is handbook.

One caution on cost

Fine-tuning has a real hidden cost beyond the training run: you now own a model version. When the base model improves, and it will, you have to decide whether to retrain. Retrieval systems inherit improvements automatically.

For a small team, that ongoing maintenance is usually the deciding argument.

---

We build retrieval systems that work against messy real-world data, and we will tell you if your problem is the rare one that genuinely wants fine-tuning. More on how we approach LLM integration, or have us build one free for three months.