How We Prevented Hallucinations with GraphRAG — NOVA Operations Log

This article documents how we structurally prevented hallucinations while building NOVA, a GraphRAG chatbot for the Vietnamese securities and legal domain. It is not about prompt engineering tricks, but about the architecture of retrieval, query generation, and validation.

The Problem

Developers often face frustration when building chatbots that must accurately answer questions about internal regulations, terms of service, and laws. Hallucinations—plausible but factually incorrect answers—occur frequently. These errors are not mere mistakes; they erode trust. Answers that fail to provide clear evidence are unusable in a business environment.

Why It Is Difficult

First, large language models do not memorize documents; they generate the next probable word. Consequently, they produce grammatically perfect answers even without actual supporting data. Second, laws and regulations form complex reference networks between clauses. Vector search alone struggles to accurately trace and connect specific provisions to the implementing decrees or subordinate regulations they reference. Third, allowing the LLM to directly generate Cypher (the database query language) introduces three simultaneous risks: security vulnerabilities, increased costs, and reduced query accuracy.

What We Did

We adopted a GraphRAG architecture combining Neo4j knowledge graphs, vector search, and the Claude API. By indexing documents along two axes—relationship-centric graphs and meaning-centric vectors—we improved retrieval precision.

To minimize LLM dependency in query generation, we introduced a 3-step strategy. 80% of queries are handled by template matching, 15% use parameterized DSL, and only the remaining 5% utilize the LLM's Text2Cypher capability. This strictly limits the proportion of queries directly written by the LLM to under 5%.

To ensure answer reliability, we applied logic to calculate a grounding score for every response. If the grounding score falls below a certain threshold, the system either withholds the answer or asks the user for clarification. Additionally, sensitive areas like investment recommendations are managed through a 3-stage blocking system involving intent classification, keyword filtering, and output filtering to ensure regulatory compliance.

Domain logic—including intents, templates, prompts, guardrails, and ontologies—was externalized into YAML domain packages. This ensures flexibility, allowing the system to be easily adapted to other industries without code changes. For the financial domain, we built a financial ontology with 17 nodes and 21 relationships by combining FinQA, FIBO, and FinDKG. To prevent performance degradation in the graph, real-time market data is not stored in the graph but managed via Redis cache with a 10-second TTL.

By the Numbers

Metric	Value	Notes
Legal Document Nodes	4,781	Neo4j LegalDocument
Legal Embeddings	20,094	multilingual-e5-large, approx. 174 minutes
Text2Cypher Dependency	< 5%	Proportion of LLM-generated queries
Unit Tests	122/122 Passed	Total test cases
Workbench API	32 Endpoints	Ontology, Pipeline, Metrics, Feedback
Vector Search Targets	119	SSV FAQ

Key Takeaway

Hallucination cannot be solved by prompt engineering alone; it must be controlled through system architecture. The core principle is limiting the LLM's role to the final 5% of the retrieval and assembly process. Readers are encouraged to randomly select 10 chatbot answers tomorrow and create a table verifying if each answer can be linked to an actual source document. Answers lacking evidence are candidates for hallucination, allowing you to immediately identify vulnerabilities in your current system.

The Problem

Why It Is Difficult

What We Did

By the Numbers

Key Takeaway

Further Reading