How production RAG works: from question to cited answer

Ask a generic AI chatbot a factual question and it will answer with total confidence — whether or not the answer is true. It cites nothing, its knowledge stops at its training date, and it has no idea which sources deserve trust. For a product whose users act on the answers, that is not a quality problem. It is a liability.

Retrieval-augmented generation (RAG) is the architecture that fixes this, and it is the pattern we used to build the answer engine behind Business111.com. This article explains how a production RAG system actually works — not the whiteboard version, the shipped one.

Retrieve, then generate

The core rule of RAG is simple: the model never answers from its own memory. Every answer is retrieved-then-generated. The system first fetches the most relevant documents from an index it controls, then asks the model to answer only from those documents, citing them as it goes. If the documents do not contain the answer, the system says so instead of guessing.

That one rule buys you three properties generic chatbots cannot offer: answers are current (the index updates continuously), answers are accurate (the model cannot invent facts that are not in the sources), and answers are verifiable (every claim traces back to a document a human can read).

A production RAG system has two halves: an offline half that builds the index, and an online half that answers questions against it.

The offline half: an index worth trusting

RAG is only as good as what it retrieves. Most of the real work is building and maintaining a curated corpus:

• Curated sources, not the open web. Content is gathered from sources chosen deliberately — government guidance, academic material, expert publications, community knowledge — and each document carries a trust score based on its source type.
• One denormalised search index. Articles, resources, and podcast transcripts all land in a single Elasticsearch index with consistent fields, so retrieval logic stays simple and fast.
• Continuous updates. New and edited content is indexed immediately, and bulk rebuilds are reproducible from the system of record. Stale answers are a retrieval problem, and this is where you solve it.
• Rich metadata. Keywords, categories, editorial boosts, and geographic tags are attached at indexing time, so the online half can rank with signals a raw text match cannot see.

The online half: four steps from question to answer

Step 1 — Understand the question. A language model rewrites the natural-language question into search keywords and extracts anything structured — in our case, UK locations for geographic filtering. This step is cached: repeat questions never pay for the model call twice.

Step 2 — Retrieve. The rewritten query runs against the index and returns the top handful of documents, ranked by relevance and editorial trust signals. Nothing generative happens here — this step is fast, cheap, and deterministic.

Step 3 — Generate, grounded. The model receives the retrieved documents and the original question, with instructions to compose a concise answer using only what the documents say, citing each claim inline. Length is capped. The model is a writer here, not an oracle.

Step 4 — Stream, with evidence. The answer streams to the user token by token over Server-Sent Events, so the page feels alive rather than frozen. Alongside the prose, the interface shows the ranked sources — the evidence rail — so a sceptical reader can check every claim.

Conversations, not just queries

Real users ask follow-ups: after an answer about registering a company, they ask what about VAT. A naive system searches for the literal words and retrieves nonsense. A production system resolves the follow-up into a standalone question first — using the recent turns of the conversation — and searches for that instead.

The resolution step is guarded deterministically: a bare yes or thanks is never sent to search, and the rewriter only sees the minimum context it needs. Conversation state lives in a fast store with rate limits attached.

What makes it production-grade

The demo version of RAG is a weekend project. The production version is defined by everything around the happy path:

• Caching at every layer. Query rewrites and repeat questions are cached, so cost does not scale linearly with traffic.
• Rate limiting. Per-user and per-anonymous-visitor daily caps keep one enthusiastic user from consuming the budget.
• Fail-soft design. If the generation model is down, search still returns ranked sources. Quality degrades; the product never breaks.
• Human judgement in the loop. Editors control the corpus, the trust scores, and the boosts. The model writes sentences; people decide what it is allowed to read.

What this means for your product

If your users ask questions your content can answer, RAG is how you give them answers they can trust — grounded, cited, current, and streamed in real time. We built exactly this for Business111.com, the AI answer engine for UK businesses, and the same architecture applies to any domain with a curated body of knowledge: internal documentation, product catalogues, regulated guidance, support content.

DevNest builds production AI systems and integrates them into the products businesses already run. Read more on our AI page at devnest.ro/ai, or see the full Business111 case study under Our Work.