Why AI Needs Facts: The Case for Layering Ontologies onto LLMs, Graph Databases, and Vector Search

Is there a role for facts in the age of LLMs? Absolutely — and it might be the missing piece that turns AI from a clever parlor trick into genuine domain expertise.

How Children Learn (And What That Tells Us About AI)

Watch a toddler figure out the world. They don’t start with definitions. Nobody hands a two-year-old a taxonomy of animals and says “memorize this.” Instead, they touch things, taste things, hear patterns in language, and slowly — through thousands of messy, unstructured interactions — they start to build an internal model of how things work.

That’s not so different from how a large language model learns. Feed it a massive corpus of text, let it absorb statistical patterns, and it develops a surprisingly rich sense of language and concepts. It can talk about medicine, law, engineering, philosophy — all with impressive fluency.

But here’s the thing about toddlers: they don’t stop there. As they grow, they start to organize. They learn that dogs and cats are both animals. That animals are living things. That living things are different from rocks. They build a structured understanding of the world — categories, hierarchies, relationships — that sits on top of all that raw experiential learning.

LLMs, for the most part, never take that second step. And that’s the problem. well ok, when an LLM is updated it is trained on books with facts, and ontologies, but really we are talking about how to teach an LLM that is already trained

Two Traditions in AI That Barely Talk to Each Other

Artificial intelligence has historically split into two camps, and they’ve spent decades largely ignoring each other.

The first camp — symbolic AI — is built on logic. It’s the world of ontologies, description logics, and knowledge graphs. You encode knowledge explicitly: “Aspirin is a subclass of NSAID. NSAIDs are a subclass of anti-inflammatory drugs. Anti-inflammatory drugs have contraindications with blood thinners.” Everything is precise, verifiable, and traceable. If the system tells you something, you can follow the chain of reasoning back to the axioms that produced it. This tradition gave us expert systems in the 1980s, the Semantic Web in the 2000s, and formal ontologies like SNOMED CT in healthcare and Gene Ontology in biology.

The second camp — statistical AI — is what powers today’s LLMs. It’s the world of neural networks, embeddings, and probability distributions. Nothing is encoded explicitly. Instead, knowledge emerges from patterns in data. This approach is extraordinarily good at handling natural language, dealing with ambiguity, and generalizing from examples. It’s also extraordinarily good at sounding confident while being wrong.

These two approaches have complementary strengths and weaknesses that fit together almost suspiciously well.

The Problem With Each Approach Alone

LLMs alone are unreliable in domains where precision matters. Ask a model about drug interactions, legal precedent, or engineering tolerances, and you’ll often get something that sounds right. Sometimes it is right. Sometimes it’s a fluent fabrication — a hallucination dressed in the language of expertise. There’s no way to know which is which without checking, and no built-in mechanism for the model to say “I derived this from axiom X and rule Y.” In medicine, finance, or law, that’s not a minor inconvenience — it’s a dealbreaker.

Ontologies alone are brittle. They can only answer questions about things that have been explicitly encoded. Ask a formal reasoner something slightly outside its modeled domain and you get silence. They can’t handle the messy, ambiguous, natural-language questions that real users ask. Building and maintaining ontologies is expensive, slow, and requires deep domain expertise. And they have no capacity for the kind of approximate, analogical reasoning that humans use constantly.

Graph databases alone give you structure and relationships, but no inference. You can traverse connections efficiently, but the database doesn’t know that if A is a subclass of B, and B has a property, then A inherits that property. It stores facts without understanding implications.

Vector databases alone give you similarity search — “find me things that are close to this concept in embedding space” — which is powerful for retrieval but has no notion of logical entailment, hierarchy, or formal correctness.

Each of these is a partial solution. Together, they’re something much more interesting.

The Layered Architecture: How the Pieces Fit

Imagine a system where an ontology provides the formal backbone — the curated, verified knowledge structure of a domain. In healthcare, that might be the relationships between diseases, symptoms, drugs, and procedures. In the project I’ve been building, DEALER, an OWL 2 Description Logic reasoner, this means parsing formal ontologies, normalizing axioms into canonical forms, and running saturation-based classification to derive every logically entailed subsumption. When DEALER tells you that concept A is subsumed by concept B, that’s not a guess — it’s a proof.

Now layer that ontology onto a graph database like rukuzu. The reasoner’s output — the full taxonomy of concepts, their equivalences, their direct parent-child relationships — gets persisted as a navigable graph. You can traverse it, query it, and use it as the structural skeleton for more complex queries. The graph gives you efficient access to the knowledge the reasoner derived.

Add a vector store to the same database. Now you can embed concept descriptions, clinical notes, legal documents — whatever unstructured text matters in your domain — and retrieve relevant content by semantic similarity. The vector layer handles the fuzziness. A user can ask a natural-language question, and the vector search finds relevant concepts even when the wording doesn’t match any formal term exactly.

Finally, put an LLM on top as the interface layer. It takes the user’s natural-language question, uses the vector store to find relevant formal concepts, queries the graph for their relationships and entailments, and synthesizes a response that’s grounded in verified knowledge rather than statistical pattern-matching alone.

The result is a system where:

• The ontology provides correctness and logical rigor
• The graph database provides structure, traversal, and persistent storage
• The vector database provides fuzzy matching and semantic retrieval
• The LLM provides natural-language understanding and generation

Each layer compensates for the others’ blind spots.

What This Looks Like in Practice

Consider a concrete scenario in healthcare. A clinician asks: “Can I prescribe ibuprofen to a patient on warfarin who has a history of GI bleeding?”

An LLM alone might give a plausible answer — or might not flag a critical interaction. An ontology alone would require the question to be formulated in formal logic. A graph query alone would return raw relationships without synthesis.

But in a layered system: the LLM parses the natural-language question and identifies the key concepts (ibuprofen, warfarin, GI bleeding). The vector store maps those to formal ontology concepts, even if the clinician used informal terms. The graph database retrieves the relevant relationships — ibuprofen is an NSAID, NSAIDs are contraindicated with anticoagulants, warfarin is an anticoagulant, NSAIDs increase GI bleeding risk. The ontology reasoner has already classified these subsumption relationships with logical certainty. The LLM then synthesizes all of this into a clear, sourced, natural-language response — and can point to exactly which axioms and relationships justify its answer.

That’s not a chatbot guessing. That’s domain expertise, synthesized.

Why Now?

This architecture wasn’t practical five years ago. Ontology reasoners were slow and academic. Graph databases were niche. Vector search was experimental. LLMs didn’t exist at useful scale.

Now all four pieces are mature enough to combine. Reasoners like DEALER can classify substantial ontologies efficiently using ELK-style saturation algorithms. Graph databases like rukuzu can store and traverse the resulting taxonomies. Vector stores are built into most modern databases. And LLMs are good enough at language to serve as a natural interface layer.

The tooling gap is closing too. Building an ontology used to require a PhD in knowledge representation. LLMs themselves can now assist with ontology construction — extracting candidate concepts and relationships from domain literature, which human experts then curate and formalize. The expensive, brittle part of symbolic AI becomes more tractable when you have a statistical model helping with the bootstrapping.

The Return of Facts

There’s something almost poetic about this convergence. For years, the AI field has been sprinting away from explicit knowledge representation toward ever-larger neural networks. And the neural networks are remarkable — but they’ve run into a wall that more parameters alone won’t fix. You can’t scale your way to factual reliability.

The answer isn’t to abandon LLMs or to retreat to pure symbolic systems. It’s to do what children eventually do: build structured knowledge on top of pattern recognition. Let the LLM handle language. Let the ontology handle truth. Let the graph handle structure. Let the vector store handle similarity.

Facts aren’t a relic of old AI. They’re the missing layer that makes new AI trustworthy.

Of course as LLMs get more powerful and larger and larger, they come pretrained in many domains, but this technique allows smaller LLMs on a mobile device, or extending an existing LLMs with knowledge from a new domain

Jonathan is the creator of DEALER, an OWL 2 EL++ Description Logic Reasoner, and rukuzu, a hybrid graph-vector database designed for knowledge representation workloads. He works at the intersection of formal reasoning and modern AI infrastructure at Loxation.

• Ontology
• Reasoning
• Vector
• Fuzzy-Logic
• Mobile
• LLM
• Graph
• Artificial Intelligence