Fine-tuning, retrieval or better prompts: which does your AI need?
Prompts change what you ask, retrieval changes what the model knows at the moment of answering, and fine-tuning changes how the model behaves. Use retrieval for facts that change, fine-tuning for consistent format, style or narrow skills, and test both against the same examples before paying for either.
What does each approach actually change?
Most disappointing AI results come from reaching for the wrong fix. A model that gets your prices wrong does not need training; it needs your price list. A model that keeps ignoring your report format may need training, after better instructions have been tried.
| Approach | What it changes, and when it fits |
|---|---|
| Better prompts | The instructions and examples sent with each request. Try this first: it is cheap, fast to change and often enough. |
| Retrieval (RAG) | What the model can see when it answers: passages fetched from your documents or systems. Right for facts that change and answers that must cite a source. |
| Fine-tuning | The model itself, adjusted on your examples. Right for consistent format, tone, classification or a narrow skill the model gets wrong despite good prompts. |
| A small trained model | A purpose-built classifier or extractor for one high-volume task. Right when speed, cost or privacy matter more than general ability. |
When is retrieval the right answer?
Use retrieval when the problem is knowledge: policies, prices, product details, contracts, anything that changes or must be quoted accurately. The system searches your approved sources, passes the relevant passages to the model and asks it to answer from them, ideally with a link to where each answer came from.
Retrieval keeps knowledge where your team can update it, respects document permissions when it is built to, and makes answers checkable. Training facts into a model does none of those things: the knowledge goes stale, and you cannot easily see where an answer came from. Our internal knowledge assistant and website chatbot pages cover this pattern in more detail.
When does fine-tuning earn its cost?
Fine-tuning pays off when the gap is behaviour, not knowledge, and you can show the gap with examples. Typical signs:
- The output must follow a strict structure, and long instructions still produce the occasional miss.
- Your categories, labels or house style are specific enough that a general model keeps getting them slightly wrong.
- One task runs at high volume, and a smaller fine-tuned model could match a larger model on it for less money and less waiting.
- Data rules require the model to run on infrastructure you control, so an open-weight model has to be brought up to standard.
Are you ready to fine-tune?
Before you fine-tune
0 of 7 done
What does a fine-tuning project involve?
- Define the task and the test. Write down what a correct output looks like and build an evaluation set from real cases, including the hard ones.
- Set a baseline. Measure the best prompt, and retrieval if relevant, on that evaluation set. This is the number training has to beat.
- Prepare the data. Collect, clean and check examples. Remove personal data the task does not need. Keep the evaluation set separate from training data.
- Train and compare. Fine-tune a hosted or open-weight model and score it on the same evaluation set, checking for regressions as well as gains.
- Deploy and monitor. Put it behind your application with logging and spot checks, and plan how often it will be retrained on new examples.
The evaluation set is the most valuable thing the project produces. It is how you know the model is better, how you catch regressions when you retrain, and how you compare a new base model next year. See how we work and how to scope an AI agent for the wider method.
What drives the cost?
The training run is often not the largest cost. Budget for these separately:
- Data work: collecting, cleaning and labelling examples, usually the biggest share of effort.
- Training compute: a hosted provider’s fine-tuning charges, or GPU time for an open-weight model.
- Running the model: per-call pricing on a hosted platform, or hosting costs if you run it yourself.
- Keeping it current: periodic retraining as your work, products or base models change.
Compare that against the cost of the alternative over a year: a larger model on every call, or the time your team spends correcting outputs. The AI agent cost guide and the ROI guide cover the arithmetic.
What about personal data?
Training data is processing under GDPR, and a model can sometimes reproduce fragments of what it was trained on. Keep personal data out of training sets unless the task genuinely needs it, record your lawful basis where it does, and check the provider’s terms for how fine-tuning data is stored and retained. An open-weight model trained and hosted on infrastructure you control keeps the data out of third-party hands. This is not legal advice; our GDPR guide covers the wider questions, and your DPO or solicitor should review the setup.
Common questions
Can fine-tuning teach a model our company knowledge?
Only partly, and not reliably. Facts learned in training go stale, cannot be updated quickly and are hard to trace back to a source. For knowledge, retrieval over your own documents is the better tool. Fine-tuning is for behaviour: format, style, labels and narrow skills.
Is fine-tuning better than a bigger model?
Sometimes. On one narrow, high-volume task, a smaller fine-tuned model can match a larger general model at lower cost and latency. On broad, varied work, the larger model usually wins. The only way to know for your task is to test both on the same evaluation set.
How long does a fine-tuning project take?
Most of the time goes on defining the task, building the evaluation set and preparing data, not on training itself. A narrow task with good existing examples can move quickly; one that needs data collected first takes longer. We scope it after seeing what data you already have.
Can we fine-tune on customer conversations?
Possibly, but treat it as personal data processing. Remove what the task does not need, check your lawful basis and privacy notice, and decide where the data and the trained model will be stored. Anonymised or synthetic examples are often enough for format and style tasks.
Bring us the task.
Tell us what happens today, which tools are involved and where it goes wrong. On a free 30-minute review, we’ll work out whether an agent, an automation or neither is the right fix, and what a sensible first release would be.
Start a project.

Book a call or send us a message. We read everything.
tash@neu.ie
Working with teams in Ireland and worldwide.
Free 30-minute AI review.