Key findings
Retrieval-Augmented Generation has become the default architecture for enterprise AI applications. The pitch is simple: instead of relying on a model’s training data, retrieve relevant documents at query time and feed them as context. It works remarkably well for straightforward question-answering. But across six production deployments at Logicbric — spanning property management, legal research, compliance screening, and ERP — we have documented the systematic failure modes where RAG alone cannot deliver production-grade results.
Finding 1: RAG fails on multi-hop reasoning
In WyseAgent, solicitors frequently ask questions that require synthesising information across multiple documents: “Does the restriction in clause 14 of the lease conflict with the planning permission granted in 2019?” RAG retrieves the lease and the planning permission separately, but the model struggles to perform the comparative analysis reliably. In our benchmark of 500 multi-hop legal queries, RAG alone achieved 61% accuracy compared to 89% when augmented with a structured reasoning chain that explicitly decomposed the question into sub-queries.
RAG retrieves information. Reasoning connects it. For any query that requires connecting information across documents, RAG alone produces unreliable results 39% of the time.
Finding 2: Retrieval quality degrades with domain specificity
Generic embedding models perform well on general-purpose text but struggle with domain-specific terminology. In Downstream ERP, terms like “turnaround” (a scheduled maintenance event in oil refining) and “cracking” (a chemical process) were being matched to semantically similar but contextually irrelevant documents. Our retrieval precision dropped from 82% on general business queries to 54% on domain-specific technical queries using off-the-shelf embeddings. Fine-tuning embeddings on domain data recovered precision to 79%.
Finding 3: Chunk size and overlap are more important than embedding model choice
We ran controlled experiments across three embedding models with five different chunking strategies. The difference between the best and worst embedding model, holding chunking constant, was 8% in retrieval precision. The difference between the best and worst chunking strategy, holding the embedding model constant, was 23%. For legal documents in WyseAgent, clause-level chunking (using document structure) outperformed fixed-size chunking by 31%. For technical manuals in Downstream ERP, section-level chunking with parent-document context retrieval proved most effective.
Finding 4: RAG cannot handle temporal reasoning
In property management, tenants and landlords frequently ask questions with temporal context: “What was the rent increase last year?” or “When does the break clause become exercisable?” RAG retrieves documents that mention these concepts but has no intrinsic understanding of time. It cannot reliably determine which version of a document is current, which clauses have been superseded by amendments, or how dates relate to each other. We addressed this by maintaining a temporal index alongside the vector store, allowing retrieval to be filtered by effective dates.
Finding 5: Hallucination rates increase with context window size
Counter-intuitively, retrieving more documents does not always improve accuracy. In our experiments, retrieving the top 3 most relevant chunks produced more accurate answers than retrieving the top 10 for 73% of queries. The additional context introduced noise, and the model would sometimes synthesise information from marginally relevant documents to produce plausible but incorrect answers. The optimal retrieval count varies by query type and should be dynamically determined by a reranking step.
Methodology
These findings are based on production data from six Logicbric deployments over 12 months, covering approximately 340,000 queries across property management (PMS), legal research (WyseAgent), compliance screening (Yezda), and industrial ERP (Downstream). We used a combination of automated evaluation (comparing against known-correct answers), expert review (domain specialists rating quality on a 5-point scale), and user feedback signals (thumbs-up/down, follow-up queries indicating confusion).
Recommendations
- Use RAG as a retrieval foundation, not a complete solution — layer reasoning, reranking, and validation on top
- Invest in domain-specific embedding fine-tuning if your accuracy on technical queries falls below 70%
- Implement structure-aware chunking that respects document semantics rather than arbitrary character counts
- Add a reranking step between retrieval and generation to filter noise and order by true relevance
- Build query decomposition for multi-hop questions — break complex queries into simpler sub-queries before retrieval
- Maintain temporal metadata on all documents and use it to filter retrieval results by relevance period
- Dynamically adjust retrieval count based on query complexity rather than using a fixed top-k