Your Local LLM Can't Remember Anything. Your Vector Database Can.

By Jonathan Borden, M.D. (jonathan@loxation.com)

How an embedded graph + vector database gives on-device AI a persistent, private, inspectable memory — without fine-tuning.

The Amnesia Problem

A large language model on your phone is powerful but amnesiac. It processes your input, produces a response, and forgets everything. The next time you ask it the same kind of question, it starts from scratch. The model weights are frozen — there's no fine-tuning happening on your device, no gradient updates, no learning in the traditional machine learning sense.

So how do you build a system that actually gets smarter over time, entirely on-device, without ever sending data to the cloud?

You give it a notebook.

The Notebook Is a Vector Database

A vector database stores data as high-dimensional numerical vectors — essentially, points in a semantic space where similar meanings land near each other. The sentence "I know a great investment opportunity, guaranteed returns" and "My friend made 300% in two months, I can show you how" end up as nearby points, even though they share almost no words. This is the magic of embedding models: they convert text into coordinates that capture meaning, not just keywords.

An HNSW (Hierarchical Navigable Small World) index on top of that data gives you sub-millisecond nearest-neighbor search. Ask "what stored examples are most similar to this new input?" and get an answer in microseconds, on-device, no network required.

Now here's the key move: the LLM validates, the vector database remembers.

Three Threats, One Architecture

To make this concrete, consider three real-world attack patterns that unfold over messaging — each with a different structure, but all detectable by the same system.

Pig butchering (sha zhu pan)

The fastest-growing fraud category globally. The attacker builds a romantic or friendship relationship over weeks, then steers the victim toward a fake investment platform. By the time the victim realizes the "platform" is controlled by the scammer, they've transferred significant money. The FBI reported $3.9 billion in losses in 2023 alone.

The attack follows a consistent arc: romantic lure → lifestyle signaling → investment introduction → platform onboarding → escalating deposits → extraction.

Grooming

A predator builds emotional dependency with a minor over time, escalating from rapport-building through isolation to exploitation. The pattern unfolds across weeks or months in a single relationship.

The arc: target selection → trust building → filling a need → isolation → desensitization → control.

Social engineering network attacks

An attacker targets multiple people in an organization to find the weakest link and extract credentials or access. Unlike the other two patterns, this one is visible in the network graph — the attacker contacts several people across departments, using different pretexts with each.

The arc: reconnaissance → pretexting → rapport → urgency/authority → credential harvesting → lateral pivot.

The Three-Layer Architecture

All three threats share a detection architecture:

Layer 1: Vector screening (every message, microseconds). Each incoming message gets converted to a vector by a small embedding model. The HNSW index checks: is this message semantically close to any known pattern from any threat category? If not, done. Cost: negligible.

Layer 2: LLM analysis (flagged conversations only, seconds). When the vector screen flags a message, the system pulls the conversation history and sends it to the local LLM. The LLM understands what no single-message comparison can: sequence, escalation, and intent across a series of messages. It returns structured confidence scores per stage:

"This looks like pig butchering in the lifestyle/investment phase. Not grooming, not SE."

Layer 3: Formal reasoning (after LLM scores, microseconds). The LLM's confidence scores feed into a lightweight ontology reasoner that combines them with structural signals — how long have you known this person? Did they contact you first? Are they in any of your groups? Have other devices flagged them? — to produce a final trust classification with a formal degree.

Here's where learning happens: when the LLM confirms a detection at high confidence, the individual messages that triggered the vector screen become new entries in the vector database. They're now part of the known pattern library. Future messages that use similar language will be caught at Layer 1 — the cheap, fast vector screen — without ever invoking the LLM.

What Gets Learned

The system ships with a curated set of example texts — call them exemplars — labeled by threat and stage. Maybe 200 for pig butchering, 200 for grooming, 200 for social engineering. Sourced from published research, law enforcement training materials, and expert-validated synthetic examples. Under 1MB total.

These curated exemplars catch the textbook patterns. But attackers don't talk like textbooks.

A curated pig butchering exemplar might say: "My mentor showed me this amazing investment platform." A real scammer on WhatsApp might say: "babe u know my uncle in singapore? he put me on this trading thing and ive been making bank lol u should try it 😂"

The embedding model sees these as broadly related — they're in the same semantic neighborhood of "personal referral to investment opportunity" — but the curated exemplar might be just far enough in vector space that the screen doesn't trigger. The system misses it.

Unless it has learned from experience.

If a previous conversation from this sender was flagged (maybe an earlier lifestyle-signaling message was close enough to a curated exemplar), and the LLM analyzed the full sequence and confirmed pig butchering at high confidence, then that WhatsApp-style message got promoted into the exemplar database. Now it's there. Next time someone uses similar casual, emoji-laden investment referral language, the vector screen catches it directly. No LLM needed.

The system didn't retrain any model. It didn't update any weights. It inserted a row in a database table. That's the learning.

Each threat learns independently

A device used by a teenager accumulates learned grooming exemplars. A device used in a corporate setting accumulates SE exemplars. A device used for dating accumulates pig butchering exemplars. Each adapts to its own threat environment.

The Graph Dimension: Patterns Invisible to Text Analysis

Here's where the three threats diverge in ways that matter for detection — and where a graph database transforms the picture.

Pig butchering has a distinctive graph signature: a new contact, no mutual connections, no shared groups, who initiated contact and quickly moves to daily messaging with lopsided initiation (they always message first early on, then the victim begins initiating as emotional investment grows).

Grooming shows a different pattern: one-to-one relationship with increasing temporal density, progressive isolation visible as the victim's other communication patterns thin out.

Social engineering has the most distinctive graph pattern of all: one sender contacting multiple people across different departments or social groups, with different pretexts to each. No amount of text analysis on a single conversation reveals this — you need to see the network.

A flat vector database can tell you "this message sounds like a social engineering pretext." A graph-backed vector database can tell you "this message sounds like a social engineering pretext, AND the sender has contacted five other people in your organization this week, AND none of them are in their contact list, AND they're asking each one about different systems."

The graph context is the difference between a flag and a conviction.

The Economics: Smarter and Cheaper Simultaneously

On a phone, LLM inference is expensive — it drains battery, generates heat, and takes seconds. Vector search is essentially free by comparison. The learning loop has a natural economic consequence:

Early on, the curated exemplar set misses many patterns. The LLM gets invoked frequently. But as learned exemplars accumulate, the vector layer handles an increasing fraction of inputs. LLM invocations decline.

The system gets smarter (catches more patterns) and cheaper (invokes the LLM less often) at the same time. This isn't a tradeoff. It's a virtuous cycle.

After a few months, the LLM is only invoked for genuinely novel patterns — language the system has never seen anything like before. Everything else is handled by the accumulated vector database at negligible cost.

For pig butchering specifically, this matters: the scam scripts evolve rapidly, with new "platforms" and new social proof tactics appearing weekly. A static detection system degrades. A learning system adapts — each confirmed detection makes the next variant easier to catch.

Why This Beats Fine-Tuning

The conventional approach to making an LLM "learn" is fine-tuning: collect training data, run gradient descent on a GPU, produce updated model weights. This works, but consider what it requires:

The vector database gives you a learning system that is transparent, reversible, incremental, private, and runs on commodity hardware. Fine-tuning gives you none of these properties.

Distributed Learning Without Sharing Data

In a peer-to-peer network, each device runs its own detection pipeline and accumulates its own learned exemplars. Device A learns to recognize pig butchering patterns in Mandarin-English code-switching. Device B learns to recognize social engineering targeting healthcare IT. Device C learns grooming patterns in teen slang. None share their exemplar data — that would leak private message content.

But they can share the conclusions. A lightweight gossip protocol transmits compact trust assessments: "I've classified device X as potentially untrustworthy, with confidence 0.72." No message content, no embeddings, no exemplar text. Just a score and a single-bit risk flag.

When device B receives this gossip and device X later contacts device B, the system starts with a prior instead of from zero. The vector screen runs with heightened sensitivity. The LLM analysis threshold is lower. The trust reasoning incorporates the peer assessment alongside local evidence.

For pig butchering, this is particularly powerful: scammers operate from call centers and contact hundreds of targets. If device A detects the scam and gossips a low trust score for that sender, devices B through Z are warned before the scammer even reaches them. The scammer's efficiency — contacting many targets with the same script — becomes a liability when the network can share conclusions.

Each device learns locally. Trust propagates globally. Data stays private.

The Formal Reasoning Layer

Raw vector distances and LLM confidence scores aren't enough for a final decision. You need to combine them with everything else you know: How long have you known this person? Are they in your contacts? Have other devices flagged them? Did you explicitly trust them?

This is where an ontology reasoner with fuzzy (graded) logic completes the picture. Each signal — vector screen hit, LLM confidence score, graph-derived behavioral pattern, peer gossip — becomes a formal assertion with a truth degree between 0.0 and 1.0:

"Device X exhibits pig butchering lifestyle signaling at degree 0.82."
"Device X is a new contact at degree 1.0."
"Device X has no mutual connections at degree 1.0."
"Device X was flagged by 2 peers at degree 0.65."

The ontology defines the rules:

If lifestyle signaling AND investment introduction, then pig butchering risk.
If pig butchering risk AND new contact AND no mutual connections, then high risk.
If high risk AND multiple peer reports, then untrusted.

The reasoner propagates degrees through these rules using a t-norm — a mathematical operator that controls how degrees combine. The conservative choice (Zadeh/min) means the weakest signal limits the conclusion: if lifestyle signaling is 0.82 but investment introduction is only 0.35, pig butchering risk is 0.35. The system doesn't jump to conclusions from partial evidence.

Crucially, the ontology also handles competing signals. If you've explicitly marked this contact as trusted (a favorite, a family member), that's a high-trust assertion that formally conflicts with the risk assertion. The reasoner resolves this tension through the same formal rules, not through an ad-hoc threshold comparison.

The General Pattern

This architecture isn't limited to messaging safety. It applies anywhere a local LLM produces judgments that should improve future performance:

Email triage: The LLM classifies messages by priority. High-confidence classifications become exemplars. Future similar emails are triaged at the vector layer without LLM.

Medical intake: The LLM assesses symptom descriptions. Confirmed assessments become exemplars. Future similar descriptions get faster initial screening.

Customer support routing: The LLM identifies intent. Confirmed intents become exemplars. Future similar queries are routed instantly.

Content moderation: The LLM evaluates flagged content. Confirmed violations become exemplars. Future similar content caught at vector layer.

Financial fraud detection: The LLM identifies suspicious transaction narratives. Confirmed fraud patterns become exemplars. Future similar narratives flagged without LLM.

In every case, the same loop: the LLM provides expensive, accurate judgment; the vector database stores the pattern cheaply; future similar inputs are handled without the LLM. The system improves by accumulating data, not by updating model weights.

The Components

The architecture requires four things:

1. An embedding model — converts text to vectors. Small (20-50MB), fast (5ms per input), runs on-device. Does not change.

2. A vector database with HNSW index — stores exemplars, answers "what's most similar?" in microseconds. Grows over time as new exemplars are promoted. This is where learning lives.

3. A local LLM — deep analysis of flagged inputs. Stateless, expensive, invoked only when the vector layer can't handle the input. Does not change.

4. A reasoning layer — combines vector scores, LLM judgments, and contextual signals into a final decision. An ontology reasoner with fuzzy logic provides formal guarantees about how signals combine. Does not change.

Only component 2 changes over time. Everything else is fixed infrastructure. Learning is entirely in the data layer.

The Sentence Version

The LLM is the teacher. The vector database is the student's notebook. The graph is the student's memory of who said what to whom. The ontology is the exam rubric. Over time, the notebook gets thicker, and the student needs the teacher less often.

---

The architecture described here is implemented using Loxation, which has an embedded graph database with built-in HNSW vector indexing, and Dealer, a fuzzy OWL 2 EL++ ontology reasoner. Both run entirely on-device — iOS, macOS, or any platform with a Rust toolchain. No cloud required.