RAG Hallucination

[Priti Mane (priti2806)]

Why does my RAG application sometimes retrieve irrelevant chunks even though the correct information exists in the document?

[AJAY TIWARI (ajtiwari1)]

Having the correct information in the knowledge base does not guarantee that the retriever will find the right chunk for a given query. The issue can originate from document ingestion, chunking, embedding generation, similarity search, ranking, or metadata filtering.

1. Common Causes

1.1 Poor Chunking Strategy

Documents are split into chunks based on character count, token count, or structural boundaries. Poor chunking can separate an answer from its heading, definition, or supporting context.

Example:

• Chunk 1: "The refund period is 30 days."

• Chunk 2: "Premium customers are eligible for special refund conditions."

If the chunks are separated incorrectly, the retriever may return incomplete information.

Solutions:

• Use semantic or structure-aware chunking.

• Preserve headings and section context.

• Tune chunk size and overlap.

• Avoid splitting tables, lists, and related paragraphs unnecessarily.

1.2 Embedding Mismatch

Embedding models convert text into vector representations. Vector similarity does not always correspond to answer relevance, particularly when queries contain domain-specific terms, abbreviations, or rare identifiers.

Example:

Query: "How can I terminate my subscription?"

Document heading: "Account Cancellation Procedure"

The correct information exists, but the embedding model may not rank it highly enough.

Solutions:

• Use an embedding model suited to your domain.

• Evaluate query and document embedding compatibility.

• Use query rewriting or multiple query variations.

• Combine semantic search with keyword search.

1.3 Incorrect Similarity Ranking

The correct chunk may exist in the vector database but rank below irrelevant chunks because other chunks receive higher similarity scores.

Example:

Rank	Retrieved chunk	Similarity score
1	Employees can request annual leave through the HR portal.	0.84
2	Leave requests require manager approval.	0.81
3	Employees receive 20 annual leave days.	0.79
4	Up to 10 unused leave days may be carried forward.	0.76

Important: Faithfulness alone cannot establish retrieval quality. A generator can faithfully use irrelevant context and still fail to answer the question.

4. Recommended Experiments

Run the same gold evaluation dataset against different retrieval configurations.

Experiment A: Baseline

• Existing chunking strategy.

• Existing embedding model.

• Current vector search implementation.

• Current top_k.

Experiment B: Chunking Optimization

Test:

• Different chunk sizes.

• Different overlap values.

• Heading-aware chunking.

• Semantic chunking.

Experiment C: Hybrid Retrieval

Combine:

• Dense vector retrieval.

• BM25 keyword retrieval.

• A fusion strategy such as Reciprocal Rank Fusion (RRF).

Experiment D: Reranking

1. Retrieve 20–50 candidate chunks.

2. Score candidates using a cross-encoder or another reranking model.

3. Select the final 5–10 chunks.

4. Compare ranking metrics and end-to-end answer quality.

These values are starting points, not universal defaults. Tune them against your dataset, latency requirements, and cost constraints.

Experiment E: Query Transformation

Test whether rewriting the query, expanding abbreviations, or generating multiple search queries improves retrieval.

For example:

Original query: "Subscription termination"

Rewritten query: "How can a customer cancel or terminate an active subscription?"

Compare the retrieval results against the same gold evidence.

5. Recommended Evaluation Workflow

1. Create a gold dataset containing questions, expected answers, source documents, and relevant gold chunks.

2. Verify that all gold chunks exist in the indexed corpus.

3. Run retrieval without generation.

4. Record the retrieved chunk IDs, ranks, scores, and metadata.

5. Calculate Recall@K, Hit Rate@K, MRR, nDCG@K, and contextual precision.

6. Compare baseline, hybrid retrieval, and reranking configurations.

7. Evaluate faithfulness and answer correctness after retrieval quality is measured.

8. Select a configuration based on retrieval quality, answer quality, latency, and cost.

Conclusion

The most useful first diagnostic is to check whether the gold chunk appears anywhere in the candidate results.

• Gold chunk missing: Investigate ingestion, chunking, embeddings, filters, and retrieval.

• Gold chunk present but ranked too low: Investigate similarity ranking, hybrid search, and reranking.

• Gold chunk retrieved but answer incorrect: Investigate context utilization and generation.

This approach separates retrieval failures from generation failures and makes RAG evaluation more systematic and measurable.