Decision guide
Retrieval-Augmented Generation — RAG — means briefly: find relevant pieces of your material, put them in the prompt, and let the model answer from that. It's the dominant way to get an LLM to talk about your products, policies or tickets without training a new model.
It's also become a catch-all for every AI wish. The result is prototypes that politely hallucinate on bad PDFs, or projects that should have been better search and metadata. We build RAG systems in production — and we say no when the prerequisites are missing.
TL;DR
Key takeaways
- RAG is right when answers must be based on your corpus, and the corpus can be chunked, updated and cited.
- If the problem is "find the right page", start with search and information architecture — not a chat UI.
- Data quality and access control decide more than model choice. Wrong permissions in retrieval is a security problem.
- Evaluation (real questions you can score) is part of the product — not an afterthought.
- Agents that "do things" are a different product from RAG that "answers with sources". Don't mix them in v1 without reason.
What RAG actually solves
An LLM doesn't know what's in your handbook, tickets or product catalogue — unless you give it the context. Fine-tuning can teach style and narrow tasks, but it's a heavy, slow answer to "answer from document X that changed yesterday". RAG is built for exactly that: fresh retrieval + generation.
The typical winning use case is internal or customer-facing Q&A over a bounded corpus: policies, runbooks, product docs, contract templates, support articles. The user asks a question, the system retrieves passages, the model formulates an answer and — importantly — can point at the sources.
The typical losing use case is "chat with the whole company" without ownership of the documents, without chunks that make sense, and without anyone willing to measure whether answers are wrong. Then you get a demo wow and a production no.
RAG vs. the alternatives
RAG
Fetch relevant context, generate an answer — the corpus can be updated continuously.
- Good for Q&A over documents and tickets
- Sources can be shown to the user
- New content is indexed without re-training
- Requires chunking, embeddings, retrieval quality
- Bad retrieval ⇒ confidently wrong answers
Other paths
Search, fine-tuning, structured extraction — or just a better UI on the data.
- Classic search: when the user must find, not get an essay
- Fine-tuning: style, classification, narrow formats — not daily doc updates
- Structured LLM calls: when output is JSON for a system, not prose for a human
- Agents/tool calls: when something must be done (create ticket, fetch order) — on top of reliable retrieval
- No AI: when a filterable table or a good FAQ tree solves the job
When is RAG the right call?
Left is the situation — right is what we typically recommend.
Your situation
Support or staff ask the same doc questions again and again
Our recommendation
RAG over a curated corpus with visible sources. Start with one domain (e.g. HR or product).
Your situation
The user needs to find a page or file, not get a generated answer
Our recommendation
Improve search, metadata and IA first. Chat can come afterwards.
Your situation
You want AI to create orders, change data or run workflows
Our recommendation
Agent with tool calls and strict guards — RAG alone is the wrong abstraction.
Your situation
Documents are messy PDFs with no owner, and nobody will clean them up
Our recommendation
Stop. Invest in content and ownership before embeddings. Otherwise you're buying hallucinations.
Your situation
Data is sensitive and role-based (own department only, own customers)
Our recommendation
RAG with retrieval that respects the same ACL as the source system — or don't build.
Your situation
You're unsure about the value
Our recommendation
Build an evaluation set of 30–50 real questions with golden answers. Prototype retrieval before you build a chat UI.
Before you build a RAG system
If these points aren't in place, the project isn't a model problem — it's a data and process problem.
- There is a bounded corpus with a named owner (not "all of Drive").
- We can list 30 questions users actually ask, with acceptable answers.
- Access rules are known: who may see which documents.
- We've agreed how new content enters (a pipeline, not manual upload forever).
- We know how we measure failure: wrong answers, missed retrieval, stale source.
- We've decided whether v1 is internal (lower risk) or customer-facing (higher cite and tone requirements).
Production is more than embeddings
In production you must re-index when documents change, handle deletion (GDPR), version prompts, rate-limit calls, and log both retrieval and answers for later analysis. "We dumped the PDFs into a vector database" is a spike — not a system.
Model choice matters for latency and cost, but rarely turns a bad corpus into a good one. Conversely, a tight corpus and good citations can make a smaller model feel more reliable than a larger model on messy data.
If you also need actions — create a case, fetch order status, book a meeting — that's an agent layer with tools, not more RAG. Retrieval can still feed the agent context; but permissions, side effects and confirmations belong in the tool layer.
Questions we get again and again
Isn't RAG obsolete now that models have longer context windows?
Longer context helps, but doesn't replace retrieval when the corpus is large, changes often, or is permissioned. Stuffing "everything" into the prompt scales poorly on cost, latency and noise. RAG (or hybrid search + rerank) remains the pragmatic choice for most company corpora.
Should we fine-tune instead?
Fine-tuning is strong for format, classification and style. It's a heavy answer for document Q&A that changes. Many teams get the most from solid RAG first, and only fine-tune if a narrow step still fails after retrieval is solid.
Which vector database should we use?
The one you can operate. Postgres with pgvector is a strong default when you already run Postgres. Dedicated vector databases make sense at very large corpora or special query patterns. The choice is secondary to chunking, metadata and evaluation.
How do we stop the model inventing answers?
You don't remove it entirely — you control it. Require citations, treat missing retrieval as a valid answer ("I didn't find anything"), keep temperature low, and score hallucinated claims against sources in your eval set. UX that shows sources also helps users trust correctly.
Can we start with ChatGPT / a custom GPT on our files?
As a spike to learn the questions: yes. As a production system for sensitive or customer-facing data: only with open eyes for data flows, permissions and lack of control. When you have proof of value, move to a setup you own — with logging, ACL and an indexing pipeline.
Considering RAG on your data?
Let's sort hype from a project that can go to production.
We'll help you settle whether the answer is RAG, better search, an agent layer — or waiting until the corpus is ready.