GenAI Still Hallucinating in 2025? Smart RAG + Knowledge Graphs May Be the Only Fix

Nabil Keith Durand

Nabil Keith Durand

— 3 min read

Smart RAG using Knowledge Graphs = Second Brain

If you ever felt GenAI delivered shallow or incorrect information - then this is currently the pioneering approach to succeed: Don't use GenAI alone, integrate it within a RAG framework. But not just basic RAG, it must be SMART.

RAG means Retrieval-Augmented Generation and put simply - its connecting GenAI to pull from your database of knowledge to craft the final answer instead of relying on murky pre-training data from last year to take a guess.

But, there is a high level of RAG failures across Fortune 500 companies because implementations often the Knowledge Management step.

Failure implementations mainly setup a basic RAG system that auto "chunks" all documents leading to poor quality GenAI responses. If poor quality chunks are the inputs for RAG, then GenAI reuses those same chunks to create poor quality answers. Garbage In = Garbage Out.

"Chunking" is a fancy word for splitting a document into group of 500 words. While this approach is fast and automated, it means each group of 500 words is a mixed bag of quality and relevance. As chunks mix many concepts together it creates more "noise" that search has to deal with - often leading to poor results.

Why? Because of a search technology called "vectorization." Each chunk is converted into a long number that floats in a database. When someone searches, the query is converted into a number and attempts to find content with similar numbers.

Smart RAG to the rescue

With Smart RAG, instead of chunking everything, all content must be curated and and converted into dense knowledge summaries of 300-500 words per topic. Then each knowledge block given metadata.

Its important to note: GenAI doesn't learn the Smart RAG content. It's seeing it for the very first time, every time it's asked. This is why using a "knowledge graph" is powerful, giving GenAI a deep dive into how all the topics in the knowledge base are interrelated.

The Knowledge Graph is is a fancy concept that means building relationships between all the knowledge blocks - like making a giant mindmap to interconnect all content. Technically, its building a robust Information Architecture and Ontology.

Smart RAG Enhancements

Enterprises are experimenting with several approaches to get even more from their Smart Rag.

  1. Priming Document - A curated reference guide (taxonomy, glossary, ontology, etc.) included in the user prompt that helps guide the Smart RAG to retrieve better results and create higher quality responses.
  2. Multi-Hop (Serial) - The user prompts AI, and Smart RAG retrieves the relevant content, but instead of generating an answer, GenAI reviews the results and makes other relevant queries for more content, and then it combines all results into a very comprehensive response.
  3. Multi-Hop (Parallel) - The user prompts AI, and GenAI determines it should split the query into several queries to create a better relevant response.
  4. User Memory - Different approaches are being used for saving content in the context window to use on follow up questions or remain in the user's profile.
  5. Evals - A prompt that tells GenAI to evaluate the end result that it meets certain conditions. Example: Is the output only from retrieved knowledge blocks?

If Smart RAG works so well, why isn't everyone deploying it?

Connecting the technologies together to make a basic RAG takes several hours to a few days. It's straight forward and can be done by many tech saavy consultants.

To unlock the full potential of Smart RAG, organizations must invest heavily in curating, organizing, and enriching their knowledge assets. This could take months or years depending on the maturity of the knowledge management program in the organization.

Only the employees who created the knowledge (domain experts) or work with it will understand what is relevant knowledge and what isn't, what knowledge is missing or is obsolete. Employees will also know the deep or hidden relationships between the knowledge blocks - adding context that isn't explicit in the original knowledge asset. A data scientist or consultant won't have a clue.

Quick and Simple Smart RAG recipe:

  1. Convert knowledge bases, books, important documents, lessons learned, best practices, procedures, etc into highly condensed knowledge blocks (about 250-500 words). Every concept, lesson, case study, story needs its own knowledge block. Pro Tip: GenAI is great at making these types of summaries.
  2. Create metadata for all knowledge blocks.
  3. Create vectors. Simply send each knowledge block to an "embedding AI" which automatically generates a long number string to save in the database.
  4. Put all knowledge blocks, metadata, and vectors into a vector database. Many options, qdrant and PostgreSQL with pgvector addon are popular choices.
  5. Create a Knowledge Graph of relationships and metadata. This requires a graph database, Neo4j is a popular choice.
  6. Create a front-end that connects API's, a python script for sending the user's queries, and the overall workflow.
💡
Pro Tip: Just copy and paste the recipe into your favorite LLM and ask for a detailed break down and plan of action. It will even generate the python scripts or any other code required to make it happen.
💡
Pro Tip: Utilize a tool such as N8N for automation, or prototyping. It is a node based workflow builder which can interconnect between LLM API's, databases, and is a great tool for cleaning up unstructured data. Many N8N nodes require coding - which LLMs are highly trained at producing. LLMs also excel at evaluating any N8N errors that arise.

Read more