Retrieval Augmented Generation (RAG) has become the standard approach for building AI applications that need accurate, up-to-date, and verifiable responses. By combining the power of large language models with external knowledge retrieval, RAG enables enterprises to build AI systems grounded in their own data. This guide covers everything you need to implement production-ready RAG systems in 2026.
Understanding RAG Architecture
RAG systems consist of three core components: a retrieval system that finds relevant documents, an embedding model that converts text to vectors, and a language model that generates responses. The key insight is that LLMs perform better when given relevant context rather than relying solely on their training data.
typescript
// Basic RAG Architecture Overview
import { OpenAIEmbeddings } from '@langchain/openai';
import { PineconeStore } from '@langchain/pinecone';
import { ChatAnthropic } from '@langchain/anthropic';
import { Pinecone } from '@pinecone-database/pinecone';
// 1. Initialize components
const embeddings = new OpenAIEmbeddings({
model: 'text-embedding-3-large',
dimensions: 1536,
});
const pinecone = new Pinecone();
const index = pinecone.index('knowledge-base');
const vectorStore = await PineconeStore.fromExistingIndex(embeddings, {
pineconeIndex: index,
namespace: 'documents',
});
const llm = new ChatAnthropic({
model: 'claude-sonnet-4-20250514',
temperature: 0,
});
// 2. RAG Pipeline
async function ragQuery(question: string): Promise<string> {
// Retrieve relevant documents
const relevantDocs = await vectorStore.similaritySearch(question, 5);
// Build context from retrieved documents
const context = relevantDocs
.map((doc, i) => `[${i + 1}] ${doc.pageContent}`)
.join('\n\n');
// Generate response with context
const response = await llm.invoke([
{
role: 'system',
content: `You are a helpful assistant. Answer questions based on the provided context. If the context doesn't contain relevant information, say so.\n\nContext:\n${context}`,
},
{
role: 'user',
content: question,
},
]);
return response.content as string;
}Document Processing and Chunking
Effective chunking is crucial for RAG performance. Chunks that are too large dilute relevance, while chunks that are too small lose context. Modern approaches use semantic chunking that respects document structure.
typescript
// Advanced Chunking Strategies
import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';
import { Document } from '@langchain/core/documents';
// Strategy 1: Recursive Character Splitting with Overlap
const recursiveSplitter = new RecursiveCharacterTextSplitter({
chunkSize: 1000,
chunkOverlap: 200,
separators: ['\n\n', '\n', '. ', ' ', ''],
});
// Strategy 2: Semantic Chunking (preserves meaning)
class SemanticChunker {
private embeddings: OpenAIEmbeddings;
private similarityThreshold = 0.85;
constructor(embeddings: OpenAIEmbeddings) {
this.embeddings = embeddings;
}
async chunk(text: string): Promise<string[]> {
// Split into sentences
const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];
// Get embeddings for each sentence
const sentenceEmbeddings = await this.embeddings.embedDocuments(sentences);
// Group sentences by semantic similarity
const chunks: string[] = [];
let currentChunk: string[] = [sentences[0]];
let currentEmbedding = sentenceEmbeddings[0];
for (let i = 1; i < sentences.length; i++) {
const similarity = this.cosineSimilarity(
currentEmbedding,
sentenceEmbeddings[i]
);
if (similarity >= this.similarityThreshold) {
currentChunk.push(sentences[i]);
} else {
chunks.push(currentChunk.join(' '));
currentChunk = [sentences[i]];
currentEmbedding = sentenceEmbeddings[i];
}
}
if (currentChunk.length > 0) {
chunks.push(currentChunk.join(' '));
}
return chunks;
}
private cosineSimilarity(a: number[], b: number[]): number {
const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
return dotProduct / (magnitudeA * magnitudeB);
}
}
// Strategy 3: Parent-Child Chunking (for better retrieval)
interface ChunkWithParent {
id: string;
content: string;
parentId: string | null;
metadata: Record<string, any>;
}
async function createParentChildChunks(
document: string,
docId: string
): Promise<ChunkWithParent[]> {
// Create large parent chunks
const parentSplitter = new RecursiveCharacterTextSplitter({
chunkSize: 2000,
chunkOverlap: 0,
});
const parentChunks = await parentSplitter.splitText(document);
// Create smaller child chunks from each parent
const childSplitter = new RecursiveCharacterTextSplitter({
chunkSize: 400,
chunkOverlap: 50,
});
const allChunks: ChunkWithParent[] = [];
for (let i = 0; i < parentChunks.length; i++) {
const parentId = `${docId}-parent-${i}`;
// Store parent chunk
allChunks.push({
id: parentId,
content: parentChunks[i],
parentId: null,
metadata: { type: 'parent', docId },
});
// Create and store child chunks
const childTexts = await childSplitter.splitText(parentChunks[i]);
childTexts.forEach((child, j) => {
allChunks.push({
id: `${docId}-child-${i}-${j}`,
content: child,
parentId,
metadata: { type: 'child', docId },
});
});
}
return allChunks;
}Advanced Retrieval Strategies
Basic similarity search often isn't enough for production systems. Advanced retrieval combines multiple strategies including hybrid search, reranking, and query transformation.
typescript
// Hybrid Search: Combining Vector + Keyword Search
import { Pinecone } from '@pinecone-database/pinecone';
interface HybridSearchResult {
id: string;
content: string;
score: number;
metadata: Record<string, any>;
}
class HybridRetriever {
private pinecone: Pinecone;
private embeddings: OpenAIEmbeddings;
private alpha = 0.7; // Weight for vector search (1-alpha for BM25)
async search(
query: string,
topK: number = 10
): Promise<HybridSearchResult[]> {
// Get query embedding
const queryEmbedding = await this.embeddings.embedQuery(query);
// Perform hybrid search (Pinecone supports sparse-dense)
const index = this.pinecone.index('knowledge-base');
const results = await index.query({
vector: queryEmbedding,
sparseVector: this.generateSparseVector(query),
topK,
includeMetadata: true,
});
return results.matches.map(match => ({
id: match.id,
content: match.metadata?.content as string,
score: match.score || 0,
metadata: match.metadata || {},
}));
}
private generateSparseVector(text: string): { indices: number[]; values: number[] } {
// Simple BM25-style sparse vector (use a proper library in production)
const tokens = text.toLowerCase().split(/\s+/);
const tokenCounts = new Map<string, number>();
tokens.forEach(token => {
tokenCounts.set(token, (tokenCounts.get(token) || 0) + 1);
});
const indices: number[] = [];
const values: number[] = [];
tokenCounts.forEach((count, token) => {
const hash = this.hashToken(token);
indices.push(hash);
values.push(count / tokens.length);
});
return { indices, values };
}
private hashToken(token: string): number {
let hash = 0;
for (let i = 0; i < token.length; i++) {
hash = ((hash << 5) - hash) + token.charCodeAt(i);
hash |= 0;
}
return Math.abs(hash) % 100000;
}
}
// Reranking with Cross-Encoder
import Anthropic from '@anthropic-ai/sdk';
class LLMReranker {
private client: Anthropic;
async rerank(
query: string,
documents: string[],
topK: number = 5
): Promise<{ index: number; score: number }[]> {
const prompt = `Given the query and documents below, rate each document's relevance from 0-10.
Query: ${query}
Documents:
${documents.map((doc, i) => `[${i}] ${doc.slice(0, 500)}`).join('\n\n')}
Return JSON array of objects with "index" and "score" fields, sorted by score descending.`;
const response = await this.client.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 1000,
messages: [{ role: 'user', content: prompt }],
});
const scores = JSON.parse(
(response.content[0] as { text: string }).text
) as { index: number; score: number }[];
return scores.slice(0, topK);
}
}
// Query Transformation: HyDE (Hypothetical Document Embeddings)
class HyDERetriever {
private llm: ChatAnthropic;
private embeddings: OpenAIEmbeddings;
private vectorStore: PineconeStore;
async search(query: string, topK: number = 5): Promise<Document[]> {
// Generate hypothetical answer
const hypotheticalDoc = await this.llm.invoke([
{
role: 'system',
content: 'Write a detailed paragraph that would answer the following question. Write as if you are writing documentation.',
},
{ role: 'user', content: query },
]);
// Use hypothetical doc for similarity search
const results = await this.vectorStore.similaritySearch(
hypotheticalDoc.content as string,
topK
);
return results;
}
}Production RAG Pipeline
A production RAG system needs caching, monitoring, error handling, and the ability to handle various document types. Here's a complete implementation.
typescript
// Production RAG System
import { Redis } from 'ioredis';
import { createHash } from 'crypto';
interface RAGConfig {
cacheEnabled: boolean;
cacheTTL: number;
maxRetries: number;
retrievalTopK: number;
rerankTopK: number;
}
interface RAGResponse {
answer: string;
sources: {
content: string;
metadata: Record<string, any>;
score: number;
}[];
cached: boolean;
latency: {
retrieval: number;
rerank: number;
generation: number;
total: number;
};
}
class ProductionRAGPipeline {
private config: RAGConfig;
private redis: Redis;
private retriever: HybridRetriever;
private reranker: LLMReranker;
private llm: ChatAnthropic;
constructor(config: RAGConfig) {
this.config = config;
this.redis = new Redis(process.env.REDIS_URL!);
this.retriever = new HybridRetriever();
this.reranker = new LLMReranker();
this.llm = new ChatAnthropic({ model: 'claude-sonnet-4-20250514' });
}
async query(question: string, userId?: string): Promise<RAGResponse> {
const startTime = Date.now();
const cacheKey = this.getCacheKey(question);
// Check cache
if (this.config.cacheEnabled) {
const cached = await this.redis.get(cacheKey);
if (cached) {
return { ...JSON.parse(cached), cached: true };
}
}
const latency = { retrieval: 0, rerank: 0, generation: 0, total: 0 };
// Step 1: Retrieve
const retrievalStart = Date.now();
const retrieved = await this.retriever.search(
question,
this.config.retrievalTopK
);
latency.retrieval = Date.now() - retrievalStart;
// Step 2: Rerank
const rerankStart = Date.now();
const reranked = await this.reranker.rerank(
question,
retrieved.map(r => r.content),
this.config.rerankTopK
);
latency.rerank = Date.now() - rerankStart;
// Get top documents after reranking
const topDocs = reranked.map(r => retrieved[r.index]);
// Step 3: Generate
const genStart = Date.now();
const context = topDocs
.map((doc, i) => `[Source ${i + 1}]\n${doc.content}`)
.join('\n\n---\n\n');
const systemPrompt = `You are a helpful AI assistant. Answer questions based on the provided sources.
Rules:
1. Only use information from the provided sources
2. Cite sources using [Source N] notation
3. If sources don't contain the answer, say "I don't have enough information"
4. Be concise but thorough
Sources:
${context}`;
const response = await this.llm.invoke([
{ role: 'system', content: systemPrompt },
{ role: 'user', content: question },
]);
latency.generation = Date.now() - genStart;
latency.total = Date.now() - startTime;
const result: RAGResponse = {
answer: response.content as string,
sources: topDocs.map((doc, i) => ({
content: doc.content,
metadata: doc.metadata,
score: reranked[i].score,
})),
cached: false,
latency,
};
// Cache result
if (this.config.cacheEnabled) {
await this.redis.setex(
cacheKey,
this.config.cacheTTL,
JSON.stringify(result)
);
}
// Log for monitoring
await this.logQuery(question, result, userId);
return result;
}
private getCacheKey(question: string): string {
const hash = createHash('sha256').update(question).digest('hex');
return `rag:query:${hash}`;
}
private async logQuery(
question: string,
result: RAGResponse,
userId?: string
): Promise<void> {
// Log to your observability platform (Datadog, etc.)
console.log(JSON.stringify({
type: 'rag_query',
timestamp: new Date().toISOString(),
userId,
question: question.slice(0, 100),
sourceCount: result.sources.length,
latency: result.latency,
cached: result.cached,
}));
}
}Evaluation and Testing
RAG systems require rigorous evaluation across multiple dimensions: retrieval quality, answer accuracy, and faithfulness to sources.
typescript
// RAG Evaluation Framework
interface EvalResult {
retrievalPrecision: number;
retrievalRecall: number;
answerRelevance: number;
faithfulness: number;
latencyP50: number;
latencyP95: number;
}
interface TestCase {
question: string;
expectedAnswer: string;
relevantDocIds: string[];
}
class RAGEvaluator {
private rag: ProductionRAGPipeline;
private llm: ChatAnthropic;
async evaluate(testCases: TestCase[]): Promise<EvalResult> {
const results = await Promise.all(
testCases.map(tc => this.evaluateSingle(tc))
);
const latencies = results.map(r => r.latency).sort((a, b) => a - b);
return {
retrievalPrecision: this.average(results.map(r => r.precision)),
retrievalRecall: this.average(results.map(r => r.recall)),
answerRelevance: this.average(results.map(r => r.relevance)),
faithfulness: this.average(results.map(r => r.faithfulness)),
latencyP50: latencies[Math.floor(latencies.length * 0.5)],
latencyP95: latencies[Math.floor(latencies.length * 0.95)],
};
}
private async evaluateSingle(testCase: TestCase) {
const response = await this.rag.query(testCase.question);
// Retrieval metrics
const retrievedIds = response.sources.map(s => s.metadata.id);
const relevantRetrieved = retrievedIds.filter(
id => testCase.relevantDocIds.includes(id)
);
const precision = relevantRetrieved.length / retrievedIds.length;
const recall = relevantRetrieved.length / testCase.relevantDocIds.length;
// Answer relevance (LLM-as-judge)
const relevanceScore = await this.scoreRelevance(
testCase.question,
response.answer,
testCase.expectedAnswer
);
// Faithfulness (is answer supported by sources?)
const faithfulnessScore = await this.scoreFaithfulness(
response.answer,
response.sources.map(s => s.content)
);
return {
precision,
recall,
relevance: relevanceScore,
faithfulness: faithfulnessScore,
latency: response.latency.total,
};
}
private async scoreRelevance(
question: string,
answer: string,
expected: string
): Promise<number> {
const response = await this.llm.invoke([
{
role: 'user',
content: `Rate how well the answer addresses the question, compared to the expected answer.
Question: ${question}
Expected Answer: ${expected}
Actual Answer: ${answer}
Score from 0-1 (just the number):`,
},
]);
return parseFloat((response.content as string).trim());
}
private async scoreFaithfulness(
answer: string,
sources: string[]
): Promise<number> {
const response = await this.llm.invoke([
{
role: 'user',
content: `Rate whether the answer is fully supported by the sources (no hallucination).
Sources:\n${sources.join('\n---\n')}
Answer: ${answer}
Score from 0-1 (1 = fully supported, 0 = contains unsupported claims):`,
},
]);
return parseFloat((response.content as string).trim());
}
private average(nums: number[]): number {
return nums.reduce((a, b) => a + b, 0) / nums.length;
}
}Best Practices
RAG Production Checklist
Chunking: Test multiple strategies; 500-1000 tokens usually works best
Embeddings: Use the latest models (text-embedding-3-large)
Retrieval: Implement hybrid search + reranking for best results
Caching: Cache embeddings and frequent queries
Monitoring: Track retrieval quality, latency, and user feedback
Testing: Build evaluation sets from real user queries
Iteration: Continuously improve based on failed queries
Frequently Asked Questions
Frequently Asked Questions
What is RAG (Retrieval Augmented Generation)?
RAG is a technique that enhances LLM responses by retrieving relevant documents from a knowledge base and including them in the prompt context. This grounds responses in factual data and reduces hallucinations.
Which vector database should I use for RAG?
It depends on your scale and requirements. Pinecone offers fully managed simplicity, Weaviate provides hybrid search, Qdrant is great for self-hosted, and pgvector works well if you're already using PostgreSQL.
How do I choose the right chunk size for my documents?
Start with 500-1000 tokens and test with your actual queries. Smaller chunks (256-512) work better for specific questions, while larger chunks (1000-1500) preserve more context for complex topics.
How do I evaluate RAG system quality?
Use metrics like Retrieval Precision/Recall, Answer Relevance, and Faithfulness. Build evaluation datasets from real user queries. Tools like RAGAS and LangSmith help automate evaluation.
Can I use RAG with Claude or GPT-4?
Yes, RAG works with any LLM. The large context windows of Claude (200K) and GPT-4 (128K) allow including many retrieved documents. The key is effective retrieval and prompt engineering.
Conclusion
RAG has matured from a research concept to an essential production pattern for enterprise AI. Success requires attention to every component: document processing, chunking strategy, retrieval quality, and response generation. The techniques in this guide provide a foundation for building accurate, scalable RAG systems.
Building an enterprise RAG system? Contact Jishu Labs for expert guidance on implementing production-ready retrieval augmented generation solutions.
SJ
About Sarah Johnson
Sarah Johnson is the CTO at Jishu Labs with expertise in AI systems architecture. She has implemented RAG solutions for enterprise clients processing millions of documents.