RAG Strategy Lab
Compare retrieval strategies on diverse datasets: from portfolio questions to recipe searches to movie analysis
What is this?
An interactive comparison of RAG (Retrieval-Augmented Generation) strategies across three diverse knowledge bases (73 portfolio docs, 381 recipes, 4K movie reviews). Compare quality, speed, and cost trade-offs.
Educational tool: Compare Basic (vector only), Hybrid (vector + keywords), Re-ranking (LLM judgment), and No RAG baseline.
Best viewed on desktop for full side-by-side comparisons and metrics tables.
β Learn about strategies below, then try live comparisons.
This reveals how RAG systems work and when each strategy excels!
My RAG Journey: Started with the AI Assistant on my homepage (Basic RAG), then built this lab to compare strategies and understand trade-offs.
Select Knowledge Base
(Loading counts...)Select Strategies to Compare
(2 selected, 2-3 allowed)What This Knowledge Base Shows73 Docs
No single strategy wins every question - This is a small, focused set (~73 documents about one person's career), so most questions work with any search strategy. The differences show up on specific kinds of questions.
Exact codes like "HL7" get blurred by meaning search, so Hybrid catches them. Questions where many documents look alike ("key technical skills") need Re-ranking to pick the right one. And a question with no single document holding the answer ("certifications") failed for every strategy until one summary document was added. Try the scenarios below and check which sources each strategy found.
When Does Each Strategy Win?
Industry best practices - click to see side-by-side comparison
When Does Each Strategy Win?
Industry best practices - click to see side-by-side comparison
Basic RAG
π― Search by meaningBest for:
- βSemantic/conceptual queries: "Tell me about healthcare experience"
- βNatural language questions: "How did she scale CM-SHARE?"
- βSmall, focused knowledge bases: Single domain, clear boundaries
Why it wins: With ~73 documents about one person's career, most questions land on the right document by meaning alone. It struggles with short codes (HL7) and with picking one document out of many similar ones (try the scenarios below).
Examples: Focused chatbots, small document collections, first prototypes
When scope is contained, semantic search alone is often sufficient. See the Embedding Visualization to see how 73 documents form clean, separated clusters.
Hybrid RAG
π Meaning + keywordsBest for:
- βExact keyword matches: Company names, acronyms (FHIR, CM-SHARE)
- βTechnical terms: "HL7", "FDA compliance"
- βLarge, diverse knowledge bases: Thousands+ documents across many topics
Why it wins: At scale (Google, enterprise wikis), semantic similarity alone creates ambiguity. "Healthcare AI" could match hundreds of documents - keyword boost helps pinpoint the exact concept you need.
Examples: Search engines that offer both keyword and vector search (Elasticsearch, OpenSearch, Azure AI Search)
Hybrid is a common default for production RAG because real questions mix concepts with exact names and codes. In this lab, the HL7 scenario shows it: meaning search misses the exact term, hybrid catches it.
Re-ranking
β‘ AI judgeBest for:
- βSubjective queries: "Best for...", "Recommend..."
- βComplex context: Multi-part questions, synthesis needed
- βHigh-stakes decisions: Worth the extra cost/latency
Why it wins: LLM understands query intent and provides nuanced relevance judgment. Cast a wide net with retrieval, then intelligently re-rank with AI.
Examples: Reranking models and services (Cohere Rerank, Voyage rerank) or an LLM scoring step like this lab uses
Use it when many documents look alike and only one holds the answer. In this lab, the skills scenario shows it: only re-ranking finds the skills summary. It adds an extra AI call, so it is slower and costs more.
No RAG
π Baseline OnlyBest for:
- βGeneral knowledge: Questions within LLM training data
- βBaseline testing: Measuring RAG value
- βDemonstrating hallucination: Shows why RAG is needed
Why it wins: It doesn't. This is for comparison only.
Examples: Proof-of-concept demos, A/B testing, educational examples
Not production-ready for domain-specific knowledge. LLMs hallucinate confidently when lacking context. Always use RAG for factual, up-to-date, or specialized information.
Quick decision: Small dataset? β Basic RAG. Large dataset or technical terms? β Hybrid RAG. High-stakes queries? β Re-ranking. Proving RAG value? β Compare with No RAG.
π¬ Research Frontier: Advanced Strategies
Techniques that shine at scale - worth exploring for large systems
π¬ Research Frontier: Advanced Strategies
Techniques that shine at scale - worth exploring for large systems
These advanced techniques add complexity but can improve retrieval quality for large-scale systems (10,000+ documents). For smaller knowledge bases like this portfolio, the strategies above are sufficient.
Multi-Query Retrieval - Generate query variations using LLM, retrieve for each, merge results. Improves recall by capturing different phrasings of the same intent.
Hypothetical Document Embeddings - Generate a hypothetical answer, embed that instead of the question. Aligns query embeddings with document space. Helps most on abstract queries.
Reciprocal Rank Fusion - Combine rankings from multiple retrieval methods using a proven formula. Simple and hard to break, because it uses ranks instead of raw scores.
Late Interaction - Token-level matching between query and documents. Balances quality of cross-encoder with speed of bi-encoder. Industry standard for large-scale re-ranking.
Recursive Summarization - Build hierarchical summaries of documents for multi-level retrieval. Best for very long documents or when you need both detail and overview.
π‘ When to explore these: If you have 10,000+ documents, see diminishing returns from basic strategies, or need sub-100ms latency at scale.
π― Try It Yourself:
Pick a scenario below. Each one has an answer key, so every strategy shows a β or β for whether it found the right source.
Then read the answers. A confident answer built on the wrong sources is the most common way RAG fails.
Behind the build
Product Evolution
Basic RAG β Hybrid Search β Re-ranking
Incremental improvements with measurable ROI
Basic RAG
Vector Search Only
Semantic similarity via embeddings. Simple, fast, and effective for most queries.
Foundation for everything
Hybrid Search
Vector + Keyword
Combines semantic similarity with keyword matching using weighted scoring (80/20 split).
Worth it for production scale
Re-ranking
LLM-based Scoring
Claude scores each retrieved document for true semantic relevance. Cast wide net, then intelligently re-rank.
For complex queries requiring judgment
π‘PM Lessons Learned
Best Strategy for Personal Sites & AI Chatbots
For portfolios, personal websites, and small knowledge bases (under 100 docs): Basic RAG is usually your best choice.
Why Basic RAG wins here:
- βSmall knowledge bases naturally cluster well - With ~73 docs about one person's career, semantic similarity creates clean separations
- βSingle domain expertise - No ambiguity when all content is healthcare PM work
- βCost-effective - 70-80% query coverage at ~$0.0003 per query vs re-ranking at 10x cost
- βFast responses - 1-2s latency keeps chatbot conversations fluid
At scale (enterprise wikis, Google Search): Hybrid and Re-ranking become essential because semantic similarity creates ambiguity across thousands of documents. But for personal sites? Basic RAG handles 70-80% of queries beautifully.
π― Bottom line: Start with Basic RAG for your portfolio chatbot. Only add complexity (Hybrid/Re-ranking) if you measure specific gaps.
Meta Discovery: Build to Debug
Plot twist: After implementing re-ranking, it performed WORSE than basic strategies (40% vs 44%). Testing revealed bugs: weak scoring prompts, insufficient context window (500 chars), and LLM giving "AI/ML Projects" higher scores than "Clinical Experience" for a clinical query.
The fix: Increased context to 1000 characters, improved prompt to emphasize query intent, added debug logging. Result? Re-ranking jumped from 40% β 92% quality (130% improvement!).
PM moment: Building features is easy. Building features that actually work well requires testing, debugging, and iteration. Your demo becomes your debugger.
RAG Implementation Guide: From Text to Multimodal
Comprehensive 10-part guide covering document preparation, chunking strategies, retrieval architecture, and production hardening. Written for PMs and developers.
RAG Tutorial by LangChain
Comprehensive guide to building RAG applications with practical examples and best practices
RAG Deep Dive by Pinecone
Technical exploration of retrieval strategies, vector databases, and semantic search fundamentals
Anthropic's RAG Documentation
Official guide on implementing RAG with Claude, including prompt engineering and context optimization
Original RAG Paper (arXiv)
Foundational research paper introducing Retrieval-Augmented Generation for knowledge-intensive NLP tasks