$ cd ../
Own RAG retrieval: BM25 + LSI embeddings + reranker — bash

user@devops:~$ cat README.md

Own RAG retrieval: BM25 + LSI embeddings + reranker

# Description

Retrieval RAG pipeline implemented from scratch in Python, with no external retrieval libraries (no Sentence-BERT, no official FAISS: the dense retrieval is our own SVD factorization). Part 1: own reference corpus of 48 ML/DS documents (pandas to world models, ~9,500 words). Part 2: sentence chunking, 4 sentences per chunk with 1-sentence overlap -> 118 chunks (52 words average, 1,860 tokens after stopwords). Part 3: BM25 with inverted postings and Robertson & Zaragoza IDF (k1=1.5, b=0.75), hand-written. Part 4: own dense embeddings: truncated SVD (32 dims) over the sparse TF-IDF, 51.0% explained variance; queries are projected with qv @ VT^T without recomputing the SVD. Part 5: hybrid fusion with Reciprocal Rank Fusion (RRF, k=60) of top-25 BM25 and top-25 dense. Part 6: GradientBoosting reranker (120 trees, depth 3) over 7 features per (query, candidate) pair: bm25, dense, sum, lexical overlap, intersection, chunk length and max. Part 7: evaluation with 50 questions with gold document. Results (seed 42, CPU ~5 s): BM25 nDCG@5 0.6257; Dense LSI 0.7514; naive RRF 0.6457 (it dilutes: fusion is not free); Hybrid+Reranker 0.8842 (+25.9 pp over BM25); out-of-sample (15 held-out queries, temporal split with no leakage) 0.7631 with Recall@5 0.8444. Part 8: 7 visualizations + demo with extractive answer and an illustrative case where BM25 fails (Pipelines) and the reranker is right (scikit-learn). Findings: our own dense beats BM25 on a topical corpus; naive fusion loses to the best single retriever, but the learned reranker gets the best of both; the dominant reranker feature is dense (0.610). 7 visualizations.

$ ./deploy.sh $ git clone

# Key features

$ Own reference corpus: 48 ML/DS documents, ~9,500 words, 1,860 tokens

$ Sentence chunking with overlap: 118 chunks of ~4 sentences (52 words average)

$ Hand-written BM25: inverted postings + Robertson & Zaragoza IDF (k1=1.5, b=0.75)

$ Own LSI embeddings: truncated SVD (32 dims) over TF-IDF, 51.0% explained variance, queries projected without recomputing the SVD

$ Hybrid fusion with Reciprocal Rank Fusion (RRF k=60) of top-25 BM25 + top-25 dense

$ GradientBoosting reranker (120 trees) over 7 features; dominant feature dense (0.610)

$ Evaluation: nDCG@5 0.6257 (BM25) -> 0.8842 (hybrid+reranker); held-out 0.7631 / Recall@5 0.8444

$ Finding: naive fusion is not free; the learned reranker combines the best of both retrievers

# Gallery

Project terminal
Own RAG retrieval: BM25 + LSI embeddings + reranker - Project terminal
Metrics of the 4 strategies
Own RAG retrieval: BM25 + LSI embeddings + reranker - Metrics of the 4 strategies
Corpus and chunk distribution
Own RAG retrieval: BM25 + LSI embeddings + reranker - Corpus and chunk distribution
LSI space in PCA 2D
Own RAG retrieval: BM25 + LSI embeddings + reranker - LSI space in PCA 2D
Recall@k per strategy
Own RAG retrieval: BM25 + LSI embeddings + reranker - Recall@k per strategy
Reranker: importance and ROC
Own RAG retrieval: BM25 + LSI embeddings + reranker - Reranker: importance and ROC
Reranker gain per query
Own RAG retrieval: BM25 + LSI embeddings + reranker - Reranker gain per query
nDCG@5 summary
Own RAG retrieval: BM25 + LSI embeddings + reranker - nDCG@5 summary

# Technologies used

Python NumPy Pandas scikit-learn Matplotlib