Every conversation with an AI agent starts from zero. It does not remember what you told it yesterday, the decision you already made twice, or the thing it learned about your work last week. It is a brilliant intern with perfect skills and total amnesia, and every morning you hire it again and re-explain everything.
I wrote before that memory, not model size, is the real unlock. This is the piece that fixes it. It is called Engram, and it is the memory layer my agents run on. Here is what is actually under the hood, because the how is the interesting part.
What it stores
Engram does not save transcripts. It saves observations. An observation is a single durable unit of knowledge: a decision, a fact, a blocker, a change, a preference. Each one carries a title, the narrative behind it, a type, an importance score, a confidence score, a status (open, resolved, stale, superseded), and a stable key so the same fact has one home instead of ten copies.
That structure is the whole game. Raw chat is noise. A clean, typed, scored observation is something a machine can retrieve, rank, and reason over later.
Capture is a filter, not a firehose
The mistake most “memory” bolt-ons make is saving everything. Save everything and you have a junk drawer, not a memory. So capture is mostly about throwing things away.
As I work, events flow into a pipeline that first asks: is this even significant. Most of it is not, and roughly two-thirds gets dropped before it ever hits the database. What survives gets normalized and checked against what already exists. Nearly identical to something stored? Skip it. Close but not the same? Merge it. Genuinely new? Keep it. The result is a memory that stays dense instead of bloating into a swamp.
The database doing four jobs at once
Under it is a single PostgreSQL database pulling quadruple duty. Relational tables for the observations themselves. A vector extension for semantic similarity. A graph extension for relationships between facts. Full-text indexes for keyword search. One store, four ways to ask it questions. Alongside it sits a dedicated search engine for fast, typo-tolerant lookups.
The embeddings, the numerical fingerprints that make semantic search work, are generated locally. No text gets shipped to an outside embedding API. That was deliberate. If the whole point is a private memory of how I think and work, it should not be quietly training someone else’s model on the way through.
Retrieval is a fusion, not a lookup
When an agent asks Engram a question, it does not run one search. It runs three at once: semantic similarity, keyword search, and typo-tolerant full-text, in parallel. Then it fuses the rankings into one list, because each backend catches things the others miss.
After the fusion comes the judgment. Results get boosted by how important the observation is, how confident it was, and how recent, with older memories decaying over time so last year’s answer does not outrank last week’s. Facts connected in the graph get a bump too. And if the query is short and vague, an LLM quietly expands it first so “the auth thing” still finds the right memory.
The relationship graph
Facts are not islands, so Engram links them. Observations connect with typed edges: this relates to that, this depends on that, this supersedes that, this contradicts that. Entities like repos, packages, and services get extracted and linked automatically, so pulling up one thing surfaces its neighbors.
The edge I care most about is contradiction. When a new fact conflicts with an old one, Engram does not silently overwrite. It marks the old one superseded, links the two, and keeps the history while serving you the current truth. The memory has a past you can walk back through, not just a present.
The part that runs without me
Storage is easy. Keeping a memory useful over months is the hard part, and it is why Engram is really a set of small agents working in the background while I do other things.
One agent sorts new observations into the right branch of a knowledge tree. Another watches for topic shifts and writes a handoff summary when the work moves on. Another keeps branch summaries short and current. An enrichment worker extracts entities, links related facts, and finds patterns. And a set of scheduled jobs quietly does the janitorial work: garbage collection, decaying the importance of things nobody touches, splitting branches that grow too big, pruning dead weight, and backing everything up.
Nobody tells it to do any of that. That is the point. A memory you have to hand-maintain is just another chore. This one curates itself.
The knowledge tree
All of it hangs on a navigable tree, the part I have rebuilt the most, now on its ninth iteration. Instead of one flat pile of thousands of memories, knowledge is organized into branches an agent can zoom into. Full detail at the branch you are working in, less as you move outward, like focusing a lens. That is how you give an agent deep context on the thing at hand without dumping the entire history into its window and drowning it.
It also writes a running record back into my notes system, so the memory is not a black box. I can open my vault and read what it has been doing.
Self-hosted, and model-agnostic on purpose
Two decisions I would not change. It runs on my own always-on machine that I reach over a private network, not on someone else’s cloud. My memory of how I work lives on hardware I control. And it is model-agnostic: it talks to Claude Code, Codex, Gemini, or anything that speaks the same protocol, exposing a few dozen tools for searching, saving, and walking the graph. I am not betting my second brain on a single vendor.
Why it was worth building
Right now Engram holds north of seven thousand observations across dozens of branches. That number is not the point. The point is that my agents no longer start from zero. They start from everything.
This lands where most of my thinking lands. The constraint is almost never the thing everyone is staring at. Everyone is watching model benchmarks. The actual bottleneck was memory, sitting off to the side, unglamorous and decisive. Memory is infrastructure. Build the layer that lets your tools remember, and they stop being clever strangers you re-brief every morning and start being something that compounds.
