Building RAG Systems That Actually Work
A practical guide to production RAG that returns grounded answers: embedding strategies, vector database choice, chunking, reranking and evaluation.
Why RAG Beats Fine-Tuning for Most Use Cases
Fine-tuning an LLM on your data bakes knowledge into the model weights. This is expensive, slow to update, and impossible to audit. RAG keeps the knowledge external in a searchable index, feeding relevant context to the LLM at query time. You can update the knowledge base without retraining, trace every answer back to source documents, and control what information the model can access.
But naive RAG—embed documents, retrieve top-K, stuff into prompt—fails in production. Building one that works requires careful attention at every stage of the pipeline.
Document Chunking Strategies
How you split documents determines whether the retrieval step finds relevant context or returns fragments that confuse the LLM.
interface Chunk {
id: string;
content: string;
metadata: {
source: string;
section: string;
pageNumber?: number;
chunkIndex: number;
tokenCount: number;
};
}
// ❌ Fixed-size chunking — splits mid-sentence, loses context
function naiveChunk(text: string, maxChars: number): string[] {
const chunks: string[] = [];
for (let i = 0; i < text.length; i += maxChars) {
chunks.push(text.slice(i, i + maxChars));
}
return chunks;
}
// ✅ Semantic chunking — respects document structure
function semanticChunk(
text: string,
options: {
maxTokens: number;
overlapTokens: number;
separators: string[];
}
): Chunk[] {
const { maxTokens, overlapTokens, separators } = options;
const chunks: Chunk[] = [];
// Split by strongest separator first
let sections = [text];
for (const separator of separators) {
sections = sections.flatMap((section) =>
section.split(separator).filter((s) => s.trim())
);
}
let currentChunk = "";
let chunkIndex = 0;
for (const section of sections) {
const sectionTokens = estimateTokens(section);
if (estimateTokens(currentChunk) + sectionTokens > maxTokens) {
if (currentChunk.trim()) {
chunks.push({
id: `chunk-${chunkIndex}`,
content: currentChunk.trim(),
metadata: {
source: "",
section: extractHeading(currentChunk),
chunkIndex,
tokenCount: estimateTokens(currentChunk),
},
});
chunkIndex++;
// Keep overlap for context continuity
const words = currentChunk.split(/\s+/);
const overlapWords = words.slice(-overlapTokens);
currentChunk = overlapWords.join(" ") + "\n" + section;
}
} else {
currentChunk += "\n" + section;
}
}
if (currentChunk.trim()) {
chunks.push({
id: `chunk-${chunkIndex}`,
content: currentChunk.trim(),
metadata: {
source: "",
section: extractHeading(currentChunk),
chunkIndex,
tokenCount: estimateTokens(currentChunk),
},
});
}
return chunks;
}The separator hierarchy matters: split on headings first, then paragraphs, then sentences. Overlap between chunks ensures that context spanning a boundary is not lost. For code documentation, treat function definitions and class boundaries as primary separators.
Embedding and Indexing
Embedding quality determines retrieval quality. The choice between embedding models matters less than how you prepare the text before embedding.
interface EmbeddingConfig {
model: string;
dimensions: number;
maxInputTokens: number;
batchSize: number;
}
class DocumentIndexer {
constructor(
private readonly embedder: EmbeddingService,
private readonly vectorStore: VectorStore,
private readonly config: EmbeddingConfig
) {}
async indexDocuments(documents: Document[]): Promise<IndexResult> {
let totalChunks = 0;
let totalTokens = 0;
for (const doc of documents) {
const chunks = semanticChunk(doc.content, {
maxTokens: this.config.maxInputTokens,
overlapTokens: 50,
separators: ["\n## ", "\n### ", "\n\n", ". "],
});
// Enrich chunks with document-level metadata
const enrichedChunks = chunks.map((chunk) => ({
...chunk,
content: this.enrichContent(chunk, doc),
metadata: {
...chunk.metadata,
source: doc.source,
documentTitle: doc.title,
lastUpdated: doc.updatedAt,
},
}));
// Batch embed for efficiency
for (let i = 0; i < enrichedChunks.length; i += this.config.batchSize) {
const batch = enrichedChunks.slice(i, i + this.config.batchSize);
const embeddings = await this.embedder.embed(
batch.map((c) => c.content)
);
await this.vectorStore.upsert(
batch.map((chunk, idx) => ({
id: chunk.id,
vector: embeddings[idx],
metadata: chunk.metadata,
content: chunk.content,
}))
);
}
totalChunks += enrichedChunks.length;
totalTokens += enrichedChunks.reduce(
(sum, c) => sum + c.metadata.tokenCount,
0
);
}
return { totalChunks, totalTokens };
}
private enrichContent(chunk: Chunk, doc: Document): string {
// Prepend document context for better embeddings
return `Document: ${doc.title}\nSection: ${chunk.metadata.section}\n\n${chunk.content}`;
}
}Retrieval and Reranking
Vector similarity search returns the K most similar chunks, but similarity is not relevance. A reranking step scores the retrieved chunks against the actual query using a cross-encoder, which is more accurate than embedding similarity alone.
interface RetrievalResult {
chunks: ScoredChunk[];
query: string;
retrievalTimeMs: number;
}
interface ScoredChunk {
chunk: Chunk;
similarityScore: number;
rerankScore?: number;
finalScore: number;
}
class RAGRetriever {
constructor(
private readonly vectorStore: VectorStore,
private readonly reranker: Reranker,
private readonly config: {
initialK: number; // Retrieve more candidates
finalK: number; // Return fewer, better results
similarityThreshold: number;
}
) {}
async retrieve(query: string): Promise<RetrievalResult> {
const start = Date.now();
// Stage 1: Broad retrieval via vector similarity
const candidates = await this.vectorStore.search(
query,
this.config.initialK
);
// Filter by minimum similarity
const filtered = candidates.filter(
(c) => c.score >= this.config.similarityThreshold
);
// Stage 2: Rerank with cross-encoder
const reranked = await this.reranker.rank(
query,
filtered.map((c) => c.content)
);
// Combine scores and take top-K
const scored: ScoredChunk[] = filtered
.map((candidate, idx) => ({
chunk: candidate.chunk,
similarityScore: candidate.score,
rerankScore: reranked[idx].score,
finalScore: 0.3 * candidate.score + 0.7 * reranked[idx].score,
}))
.sort((a, b) => b.finalScore - a.finalScore)
.slice(0, this.config.finalK);
return {
chunks: scored,
query,
retrievalTimeMs: Date.now() - start,
};
}
}Prompt Construction and Generation
The final prompt must clearly separate retrieved context from the user's question. The LLM needs instructions about how to handle conflicting sources, missing information, and the boundaries of what it should answer.
function buildRAGPrompt(
query: string,
retrievedChunks: ScoredChunk[],
conversationHistory: Message[]
): string {
const context = retrievedChunks
.map(
(chunk, idx) =>
`[Source ${idx + 1}: ${chunk.chunk.metadata.source}]\n${chunk.chunk.content}`
)
.join("\n\n---\n\n");
return `You are a helpful assistant. Answer the user's question based ONLY on the provided context. If the context does not contain enough information to answer, say so explicitly. Do not make up information.
If multiple sources contradict each other, note the discrepancy and present both perspectives.
## Context
${context}
## Conversation History
${conversationHistory.map((m) => `${m.role}: ${m.content}`).join("\n")}
## Current Question
${query}
## Instructions
- Cite sources using [Source N] notation
- If the answer requires information not in the context, state what is missing
- Be concise and direct`;
}Evaluation and Quality Metrics
You cannot improve what you do not measure. RAG systems need evaluation along two axes: retrieval quality (did the system find the right chunks?) and generation quality (did the LLM produce a correct, grounded answer?).
interface RAGEvaluation {
retrievalMetrics: {
precision: number; // Relevant chunks / retrieved chunks
recall: number; // Retrieved relevant / total relevant
mrr: number; // Mean Reciprocal Rank
};
generationMetrics: {
faithfulness: number; // Is every claim supported by context?
relevance: number; // Does the answer address the question?
completeness: number; // Are all relevant aspects covered?
};
}
async function evaluateRAG(
testCases: Array<{
query: string;
expectedChunkIds: string[];
expectedAnswer: string;
}>,
retriever: RAGRetriever,
generator: RAGGenerator
): Promise<RAGEvaluation> {
let totalPrecision = 0;
let totalRecall = 0;
let totalMRR = 0;
for (const testCase of testCases) {
const result = await retriever.retrieve(testCase.query);
const retrievedIds = result.chunks.map((c) => c.chunk.id);
const relevant = retrievedIds.filter((id) =>
testCase.expectedChunkIds.includes(id)
);
totalPrecision += relevant.length / retrievedIds.length;
totalRecall += relevant.length / testCase.expectedChunkIds.length;
// MRR: position of first relevant result
const firstRelevantIdx = retrievedIds.findIndex((id) =>
testCase.expectedChunkIds.includes(id)
);
totalMRR += firstRelevantIdx >= 0 ? 1 / (firstRelevantIdx + 1) : 0;
}
const n = testCases.length;
return {
retrievalMetrics: {
precision: totalPrecision / n,
recall: totalRecall / n,
mrr: totalMRR / n,
},
generationMetrics: {
faithfulness: 0, // Requires LLM-as-judge evaluation
relevance: 0,
completeness: 0,
},
};
}Key Takeaways
Production RAG systems fail when any single stage is weak. Chunking determines whether relevant context is retrievable. Embedding quality depends more on input preparation than model choice. Reranking is the single highest-impact addition—it converts broad similarity into precise relevance.
Build the evaluation harness before optimizing anything else. Without ground-truth test cases measuring retrieval precision, recall, and generation faithfulness, every change is guesswork. Start with a small, high-quality test set and expand as failure modes emerge.
The prompt is the contract between retrieval and generation. It must clearly delineate context from instructions, tell the model when to refuse, and require source citations. A well-structured prompt with mediocre retrieval outperforms a poor prompt with excellent retrieval every time.
