# RAG or fine-tuning? What your company’s AI assistant actually needs

> Most assistants that answer from company documents need retrieval, not a trained model. The difference in plain words, and how to tell which you need.

- Author: Prayag Dalal (https://shipfast.online/about/)
- Published: 2026-10-07
- Canonical: https://shipfast.online/blog/rag-vs-fine-tuning/

## Key takeaways

- Most company assistants need RAG (retrieval at question time), not a trained model.
- RAG keeps answers up to date and shows sources; documents can be added or removed without retraining.
- Fine-tuning teaches a style, format or narrow task; it is a poor way to store facts that change.
- Clean documents, good retrieval, citations and an evaluation set matter more than the model.

When a team says "we want to train an AI on our documents", they usually mean something simpler, cheaper and easier to keep up to date: an assistant that looks things up in their documents before it answers. That approach is called retrieval-augmented generation, or RAG.

## What RAG does

1. Your documents (PDFs, docs, wiki pages, help articles) are split into short passages.
2. Each passage is turned into a vector, a list of numbers that captures its meaning, and stored in an index.
3. When someone asks a question, the most relevant passages are retrieved from that index.
4. A large language model writes the answer using only those passages, and links back to them.

Nothing about the model itself changes. It reads your documents at question time, the way a new colleague would check the handbook.

## What fine-tuning does

Fine-tuning changes the model's weights by training it on examples. It's good at teaching a model a style, a format, or a narrow task it does over and over. It is not a good way to store facts that change, because updating the facts means training again.

## Which one you need

Choose RAG when:

- the answers live in documents that change over time
- people need to see where an answer came from
- you want to add or remove documents without retraining anything

Consider fine-tuning when:

- you need a very specific output format or tone, consistently
- the task is narrow and repetitive, with many good examples available
- retrieval alone is already working and you want to refine behaviour

For most company assistants, support bots and internal knowledge search, RAG is the right starting point. Fine-tuning can come later, if it's needed at all.

## What makes a RAG assistant good

The model matters less than people expect. What matters more:

- **Clean source documents.** Outdated or contradictory pages produce contradictory answers.
- **Good retrieval.** Passage size, search method and ranking decide whether the right text reaches the model.
- **Citations.** Every answer should show its sources, so people can check it.
- **An evaluation set.** A list of real questions with known good answers, re-run after every change, so quality goes up instead of drifting.

If you're weighing an assistant for your team or your customers, see how I build [RAG systems and AI assistants](https://shipfast.online/services/rag-development/).

---

Book a 30-minute call: https://cal.com/prayagdalal/30min