"Should we fine-tune or use RAG?" is the question we hear most from teams moving past the prototype phase. It's usually the wrong question — the two techniques solve different problems, and the teams that struggle are the ones using either to do the other's job. Here's the decision framework we use, with the eval data behind it.
They optimize different things
RAG changes what the model knows at request time. It retrieves relevant documents and puts them in context — perfect for knowledge that is voluminous, changes often or must be cited. Fine-tuning changes how the model behaves. It bakes in tone, format, domain vocabulary and task-specific reflexes — things you'd otherwise burn prompt tokens re-explaining on every call.
- Facts that update daily, per-customer data, compliance-sensitive citations → RAG.
- Consistent output schema, brand voice, niche jargon, shorter prompts at scale → fine-tuning.
- "The model doesn't know our docs" is a retrieval problem. "The model won't stop being chatty in our JSON" is a behavior problem.
The cost curves cross
RAG is cheap to start — an index and an embedding pipeline — but every request pays a context tax: retrieved chunks are input tokens, forever. Fine-tuning is expensive up front (data curation dominates; training itself is the cheap part) but then reduces per-request cost, because a tuned model needs shorter prompts. In our modeling, the curves cross around 2–5 million requests per month: below that, RAG alone is almost always the right economics; above it, tuning the behavior and retrieving only fresh facts wins.
What our evals showed
We built a 1,200-case eval from real support and analytics workloads and ran four configurations of tapotik-2-pro: base, RAG-only, fine-tune-only, and hybrid. Grading was blind, with a rubric-scoring judge audited by humans on a 10% sample.
const results = await evals.run({
suite: "support-kb-v3", // 1,200 graded cases
configs: [
{ name: "base", model: "tapotik-2-pro" },
{ name: "rag", model: "tapotik-2-pro", retrieval: { index: "kb_prod", top_k: 6 } },
{ name: "tuned", model: "ft:tapotik-2-pro:support-v2" },
{ name: "hybrid", model: "ft:tapotik-2-pro:support-v2", retrieval: { index: "kb_prod", top_k: 4 } },
],
judge: { model: "tapotik-2-pro", rubric: "factuality+format" },
});
// factuality: base 61% | rag 89% | tuned 71% | hybrid 93%
// format: base 74% | rag 76% | tuned 96% | hybrid 97%The pattern held across suites: RAG dominated factuality, tuning dominated format and tone, and the hybrid beat both on every axis while using 38% fewer input tokens than RAG-only — the tuned model needed less instruction and fewer retrieved chunks to get the same answer right.
Fine-tuning teaches the model how to behave. Retrieval tells it what's true today. Most production systems eventually need both — just not on day one.
The framework, in one paragraph
Start with RAG — it's debuggable, updatable and cheap at low volume. Add fine-tuning when one of three triggers fires: your system prompt has grown past ~2K tokens of behavioral instruction, your output format failures are user-visible, or your request volume makes the per-token math favor shorter prompts. And before any of it, build the eval suite — without one, both techniques are just expensive guessing.