RAG vs. fine-tuning vs. a better prompt: a decision guide

Almost every AI project we get pulled into starts with a team already convinced they need to fine-tune a model. Sometimes that's right. Usually the actual problem is something a prompt change or a retrieval layer solves in a week, for a fraction of the cost and none of the maintenance burden. The order we check things in matters.

First: is this actually a prompting problem?

A surprising share of "the AI got it wrong" complaints turn out to be "the prompt didn't give the model what it needed to know, in a form it could use." Before reaching for infrastructure, we push on the prompt itself — more specific instructions, a couple of worked examples, an explicit output format — and measure whether the failure rate actually improves. It's cheap to test and it's embarrassing to skip.

Reach for RAG when the gap is missing knowledge

Retrieval-augmented generation is the right tool when the model already knows how to answer the kind of question you're asking — it just doesn't have your specific data. A support bot answering from your documentation, internal search over a wiki, contract Q&A over a document set: all knowledge gaps, not skill gaps. The real cost isn't the vector database, it's the chunking strategy and the discipline to have the model say "I don't know" instead of confidently guessing when retrieval comes back empty. That failure mode — a wrong answer stated as fact — is the one that actually damages trust in the product.

Reach for fine-tuning when the gap is missing behavior

Fine-tuning earns its cost when you need consistent tone, format, or task execution across thousands of calls in a way prompting can't reliably hold onto, or when you want a much smaller and cheaper model to match a skill a larger general model already has. The honest tradeoff: it needs curated training data and a real evaluation harness, and unlike RAG it goes stale as your product changes, because it isn't reading your live data — it baked a snapshot of behavior into the weights.

They're not competing options

Most production systems we ship end up doing a little of both: RAG supplies current facts, and either a well-built system prompt or a light fine-tune shapes how the model uses them. The mistake is treating this as an either/or decision made once at the start of a project, rather than adding each piece only once a concrete evaluation shows the simpler option isn't enough.

The one thing to never skip

Whichever route you take, build an evaluation set of real questions with known-good answers before you ship, and rerun it after every prompt, retrieval, or model change. Without it, "we improved the AI" is just a feeling — and the first time you find out it's wrong is a support ticket from a customer.