Building a Production RAG Chatbot: 7 Lessons from Shipping an HR & Payroll Assistant

I build an AI assistant that answers HR and payroll questions from company policy documents. In that domain a hallucination is not embarrassing — it miscalculates someone's salary. These are the lessons that survived contact with production.

1. Retrieval quality is the product

Everyone obsesses over the model. The model is fine. What kills RAG systems is retrieval: the right passage never reaches the context window, and the model improvises around the gap.

Spend your time on chunking strategy, embedding quality, and testing retrieval in isolation — given this question, did the right passages come back? If retrieval fails, nothing downstream can save you.

2. Citations are a feature, not decoration

Every answer in our system points back to the document and section that says so. This does two jobs. Users can verify — which builds the trust that decides whether the tool gets used at all. And it disciplines the generation: a model forced to cite is a model with less room to invent.

3. Refusal behaviour is where the engineering lives

The hardest part of a production RAG system is teaching it to say "I don't know". When retrieval comes back empty or weak, the system must refuse — visibly, gracefully, and every single time. An assistant that answers everything is lying about something.

We treat refusals as first-class outputs: they get logged, reviewed, and used to find gaps in the document corpus. A refusal is data.

4. Evaluate on wrong-answer rate, not vibes

Demos lie. We measure against a fixed set of real questions with known correct answers, and the metric that matters is the wrong-answer rate — how often the bot answers confidently and incorrectly. That number decides whether the system ships, not how impressive the demo felt.

Build the eval harness before anyone types "hello". Retrofitting one after stakeholders have seen the shiny demo is a political fight you will lose.

5. The corpus is never ready

Policy documents contradict each other. They go stale. Two PDFs answer the same question differently depending on the year they were written. A RAG project is secretly a document-hygiene project, and the sooner you accept that, the sooner the answers get good. Assign an owner for the corpus or watch quality decay.

6. Guardrails belong in the system, not the prompt

"Please only answer from the context" is a suggestion, not a control. Real guardrails are structural: retrieval-empty checks that short-circuit before generation, answer validation against retrieved text, and domain boundaries enforced in code. The prompt is the last line of defence, not the first.

7. Users ask questions you did not design for

Production traffic is humbling. People ask compound questions, questions in the wrong language, questions about documents that do not exist, and questions where the honest answer is "call HR". Log everything (within privacy rules), review weekly, and let real usage — not your imagination — drive the roadmap.

The stack matters less than the discipline

Ours happens to be embeddings + vector retrieval + an LLM with grounded prompting, evaluated continuously. You can build the same thing on half a dozen stacks. What you cannot swap in later is the discipline: retrieval-first thinking, citations, refusal behaviour, and honest evals.

I write more about shipping AI systems that survive production — the portfolio has the receipts, including this site's own grounded concierge. Building something like this? Bring the real problem — corpus, users, and what a wrong answer costs you.