Hybrid RAG Search with BM25 and Rank Fusion in Python

A user asks how to resolve ERR-104, but the retriever returns a general troubleshooting page instead of the document that names that error. Another user describes the same problem without its code; a literal keyword search misses the right page. These are different retrieval failures. Hybrid search combines lexical and semantic candidates so you can inspect and measure both signals.

This article implements a small BM25 baseline and reciprocal rank fusion (RRF) in Python. It complements RAG with LangChain, which covers the pipeline, and offline RAG evaluation, which covers evidence availability. The new decision here is how to combine candidate rankings before passing context to the generator.

Decide what each retriever should contribute

BM25 uses term frequency, document frequency and document length. It is useful for explicit identifiers, product names and words that must appear. Dense retrieval compares embeddings and can find paraphrases without the same surface wording. Neither signal establishes that a passage answers the question.

For example, keep ERR-104 distinct from ERR-105, and an API version v2.1 distinct from v2.0. Tokenization is part of the retrieval contract. Splitting every identifier at punctuation can erase the distinction you were trying to preserve.

Elasticsearch's BM25 settings describe k1 as term-frequency saturation and b as document-length normalization. The defaults used below, 1.2 and 0.75, are a reproducible starting point, not tuned values for your corpus.

Run an inspectable lexical baseline

The example needs Python 3.11 and no third-party packages, server, model download or API key. Save the following blocks in order as hybrid_search.py. This implementation scans a tiny corpus; use an indexed retrieval engine for production-sized collections.

import math
import re
import unicodedata
from collections import Counter, defaultdict


def tokens(text: str) -> list[str]:
    normalized = unicodedata.normalize("NFKC", text).casefold()
    return re.findall(r"\w+(?:[-.]\w+)*", normalized)


def bm25(query: str, documents: dict[str, str], limit: int = 3) -> list[str]:
    if type(limit) is not int or limit <= 0:
        raise ValueError("limit must be a positive integer")
    if not documents:
        return []
    counts = {key: Counter(tokens(text)) for key, text in documents.items()}
    lengths = {key: sum(count.values()) for key, count in counts.items()}
    average_length = sum(lengths.values()) / len(lengths)
    if average_length == 0:
        return []
    terms = set(tokens(query))
    document_frequency = {
        term: sum(term in count for count in counts.values()) for term in terms
    }
    scores = {}
    k1, b = 1.2, 0.75
    for key, count in counts.items():
        score = 0.0
        for term in terms:
            frequency = count[term]
            if frequency == 0:
                continue
            df = document_frequency[term]
            idf = math.log(1 + (len(documents) - df + 0.5) / (df + 0.5))
            denominator = frequency + k1 * (
                1 - b + b * lengths[key] / average_length
            )
            score += idf * frequency * (k1 + 1) / denominator
        if score > 0:
            scores[key] = score
    return sorted(scores, key=lambda key: (-scores[key], key))[:limit]

The token pattern preserves dotted versions and hyphenated codes, while Unicode normalization and case folding handle straightforward formatting differences. It does not implement stemming, synonyms, language-specific analyzers or special treatment for names such as C++. Test representative queries before adopting a tokenizer. Spanish contraseña and English password still need a multilingual semantic model or an explicit translation/synonym strategy.

The formula is an educational BM25 variant. It follows the term weighting described in Elastic's explanation of similarity, but is not a byte-for-byte replacement for Lucene analyzers and index norms. Results with zero lexical score are omitted: an unknown identifier should not produce a fabricated lexical match.

Fuse ranks instead of adding incomparable raw scores

A BM25 score and an embedding similarity have different scales. Adding them with an arbitrary weight can let one retriever dominate because of its numeric range. RRF adds a contribution based on position in each list:

score(document) = sum(1 / (smoothing + rank))

In this example ranks start at one, both lists have equal weight, and smoothing=60. The Elasticsearch RRF reference documents rank-based fusion. Other engines may use different constants or rank conventions; Qdrant's hybrid query reference currently documents a different default. Configure these explicitly when comparing implementations.

def rrf(*rankings: list[str], limit: int = 3,
        smoothing: int = 60) -> list[str]:
    if type(limit) is not int or limit <= 0:
        raise ValueError("limit must be a positive integer")
    if type(smoothing) is not int or smoothing < 0:
        raise ValueError("smoothing must be a non-negative integer")
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        if len(ranking) != len(set(ranking)):
            raise ValueError("A retriever returned duplicate chunk IDs")
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] += 1 / (smoothing + rank)
    return sorted(scores, key=lambda key: (-scores[key], key))[:limit]


if __name__ == "__main__":
    documents = {
        "general": "Troubleshooting guide for failed account login.",
        "reset": "Reset an expired password using account recovery.",
        "err-104": "ERR-104 means the login token expired. Request a new token.",
        "err-105": "ERR-105 means the account is locked. Contact support.",
    }
    lexical = bm25("ERR-104", documents)
    # Fixed fixture: replace with ranked chunk IDs from your dense retriever.
    semantic = ["general", "reset", "err-104"]
    print("lexical:", lexical)
    print("fused:", rrf(lexical, semantic))

Run python hybrid_search.py. It prints:

lexical: ['err-104']
fused: ['err-104', 'general', 'reset']

The semantic ranking is a deliberately supplied fixture, not the output of an embedding model. It makes the combination reproducible and exposes the failure: the relevant error-code document was present but ranked third. Fusion promotes it because both lists retrieve it. This is a demonstration of rank arithmetic, not measured evidence that hybrid search improves your RAG system.

Duplicate IDs within one list raise an error so repeated chunks cannot inflate a score. The same ID across two different lists is expected and receives both contributions. Ties break by ID for repeatable experiments; a production tie-break policy should also be explicit.

Candidate depth comes before final context size

Separate three budgets: candidates per retriever, candidates passed to a reranker, and passages passed to the LLM. Retrieving 30 candidates from each signal and returning five fused passages is different from retrieving five from each. Those numbers are examples, not recommendations for every index.

If the relevant document is missing from both candidate lists, RRF cannot recover it. A reranker also cannot recover a document it never receives. Increasing the final context window alone does not fix that missing-candidate failure.

Use a stable chunk ID across the two indexes and version the mapping when rechunking. Apply the same authorized document scope before candidate retrieval in each branch. Filtering only the fused list can expose unauthorized data in intermediate stages and fill the budget with inaccessible candidates. Neither RRF nor a relevance score supplies authorization.

Evaluate the change without moving the goalposts

Create reviewed cases for exact codes, ordinary terminology, paraphrases, Spanish/English queries, and questions the corpus cannot answer. Store the relevant chunk IDs and the expected abstention behavior. Keep a development split for tuning and a separate fixed split for comparison.

Compare lexical-only, semantic-only and fused rankings at the same final k. Also report candidate recall before fusion, because it separates retrieval misses from ranking misses. Record latency, empty lexical results, total candidate count, repeated document count and the context budget. A single aggregate recall can hide a regression in rare identifiers.

The executable example was checked for exact identifiers, zero-match queries, empty documents, duplicate IDs, stable fusion output and a missing candidate. It contains no production corpus, embedding run or accuracy benchmark. Use the offline evaluation harness to attach real reviewed labels before choosing a configuration.

If hybrid candidates solve your observed failure, add a reranker only when labeled cases show that ordering still limits the answer. If the correct evidence already reaches the generator, work on context selection and provenance instead. The next useful change is the one that addresses the measured failure, not the longest retrieval pipeline.