Five calls in one week. All different people, all wanting to chat about the same thing: memory for AI agents.
I’ve been noodling on this for almost four months. Researching, building, tearing things apart, rebuilding. And obviously I’m not the only one feeling the urgency. Memory is the topic I’m asked about most, and there’s a reason: without it, everything else about agents falls apart.
So I wanted to document where I am after four months of hands-on work and a lot of research. If it’s useful to somebody else, great. If somebody points me in a better direction, even better.
The bigger picture
Before I get into memory specifically, I want to lay out the frame I’m working inside.
A lot of people are still approaching LLMs the way they approached Google search. You type something in, you get something back, you move on. That’s fine for quick lookups. But the shift I keep coming back to is treating your LLM more like a coworker. Not in an anthropomorphization way (that has its own downsides), but practically: an entity that can learn, retain knowledge, update it, do research, and take action based on all of that context.
That’s what an agent actually is. And if you look at the latest releases by Anthropic, Google and OpenAI you can see that’s also their focus. Everybody is pushing in that direction. When I think from first principles about what it takes to get there, I land on four building blocks:
Memory and self-learning — the agent knows things about you, your work, your preferences, its own previous work, and retains that across sessions
Deep research — the agent finds information, evaluates it, and integrates it. Especially with how fast things move today, llm-training knowledge alone is inadequate
Autonomous operation — putting it all together so the agent can work independently, iteratively
Containers — the agent needs isolated environments to operate safely. This is harder than it sounds because there’s real tension between access (to carry out tasks) and isolation (to preserve security)
This piece is about the first one.
Memory is the foundation. Without it, the other three can’t function. An agent that forgets everything between sessions can’t learn. An agent that can’t learn can’t improve. And an agent that can’t improve isn’t really autonomous. It’s just a tool you have to drive manually every time.
I’m planning to write about each of these building blocks as I work through them. If that sounds useful, subscribe to follow along. But for now: memory.
Why memory matters
I’m in the middle of a relocation. My family (partner and toddler, family of three) is considering a few cities in Canada and Europe. Lots of research across each city: neighborhoods, housing costs, schools, parks, whether you need a car, cost of living. All happening in parallel, across many chat sessions over days and shared between my wife and I.
One afternoon I asked my AI agent to look up average apartment rental costs in Calgary. I needed the numbers for our monthly expense budget. The agent went off to do web searches, and since these things take a few minutes, I switched to another tab and kept working.
When I came back, the numbers looked great. Lower than I’d expected. I knew Calgary was cheaper, but this was a lot cheaper than I thought. I was relieved. Plugged the numbers into my budget spreadsheet, started calculating what that meant for our total monthly costs.
Then I looked more carefully. The agent had averaged all rentals in Calgary. Studios, one-bedrooms, everything. We need a two-bedroom. I’d discussed the family situation in a previous chat, but this was a different session. I forgot to re-specify it, and the agent had no memory of that conversation.
Every number was wrong. The budget I’d just built was fiction.
The worst part wasn’t the wasted time or the API costs. It was the false sense of security. The numbers had looked perfectly reasonable, plausible enough that I’d built plans on top of them without questioning. When you think your AI is factoring in your criteria and it’s not, you get a completely misleading sense of confidence in the results.
You could argue the failure was on me, if nothing else for not validating the research sources. But if we’re honest with ourselves, verifying sources sits right next to reading the contracts before hitting “I agree” during an software install process.
After that, something shifted. Every subsequent search felt suspect. I started second-guessing, over-specifying everything, carrying this mental overhead of “did I remember to tell it about the family?” across every single interaction. The tool stopped feeling like a help and started feeling like something I had to babysit.
And this same context, family of three, was needed across all cities, across every research thread and shared between my wife and I. Without memory, each thread was a fresh opportunity to miss it.
This is why memory matters. Not because it’s nice to remember that I prefer dark mode. That’s trivial. Memory matters because without it, there’s no foundation for trust. Without trust, you can’t delegate. And if you can’t delegate, the agent isn’t really an agent. It’s just a fancy search box.
How to organize memory
Most people’s first instinct when thinking about agent memory is “short-term versus long-term.” That’s a start, but it’s too coarse. It conflates how long you keep something with what kind of thing it is, and that distinction matters when you’re trying to categorize, compress, retrieve, and strengthen memories over time.
A better model comes from cognitive psychology. In 1972, Endel Tulving proposed a taxonomy of human memory that maps surprisingly well to AI agent design.
Semantic memory is facts and knowledge. Context-free, general information: “Spike codes in TypeScript,” “Calgary is in Alberta,” “a family of three needs at least two bedrooms.” Tulving called this a “mental thesaurus.” In agent terms: extracted facts, user preferences, general knowledge.
Episodic memory is experiences and events. Time-stamped, personal: “last time we deployed on Friday, it broke,” “the Calgary apartment search returned wrong results because we forgot the family criteria.” In agent terms: interaction histories, task trajectories, past sessions. This is what enables case-based reasoning: remembering not just the fact, but the context in which you learned it.
Procedural memory is skills and processes. How-to knowledge, performed almost automatically: “always run tests before committing,” “when researching a new city, start with cost of living then drill into neighborhoods.” In agent terms: tool-use patterns, workflow templates, behavioral habits.
What’s interesting is how these types interact. Episodic memories consolidate into semantic ones: you have the same experience three times and it becomes a general fact. Semantic knowledge consolidates into procedural knowledge: you know something so well it becomes automatic behavior. These consolidation pathways are how an agent learns, not just remembers.
Two pieces of metadata matter for every memory, and most implementations miss one or both.
First: source confidence. Where did this memory come from? A direct user correction (“No, always do it this way”) is authoritative, high trust. An observed pattern repeated three or more times is trusted. A single-occurrence inference is untrusted. These trust levels affect how aggressively you apply a memory and how you resolve conflicts.
Second: freshness. When was this recorded? Some memories have short lifespans. The cost of apartments in Calgary right now is different from the cost six months ago. Date metadata lets you implement natural memory decay. Recent things feel more present, older things fade. This is what Ebbinghaus described with his forgetting curves in the 1880s, and modern implementations like MemoryBank use the same approach: refresh memories when they’re retrieved, prune the ones that fall below a salience threshold.
One more thing: memory should have weights. If I express the same preference in three different contexts, say preferring short functions in code, that preference should get stronger, not just get recorded three times as separate entries. Most systems treat memories as flat lists. They shouldn’t. Hong and He formalized this in 2025 as Weighted Memory Retrieval: a combination of recency decay, importance scoring, and relevance. It matches how human memory works, and how LLMs get trained. Strengthening links. The parallel isn’t accidental.
Diary, decision logs, and what memory actually means
There’s a common confusion I want to address. People treat “memory” as synonymous with “facts I remember.” But when you think about what actually matters for an effective agent, or an effective colleague, it’s broader than that.
Consider the apartment search again. The fact “average 2-bedroom rent in Calgary is $X/month”? That’s semantic memory. But how did I get there? The back-and-forth with the agent, the specific search criteria I pushed for, the fact that “dens” are a common listing type and seemed like an option but turned out not to work for us? That’s something different. That’s a diary. A narrative of what happened, how the thinking evolved, what decisions were made along the way based on what facts and preferences.
Some people implement diaries as ordered lists of memories. I disagree. A diary is a fundamentally different thing. It captures the story, the path, the why behind the what. The decision trail specifically (what was decided, what alternatives were considered, what was decided against) is what software engineers call an Architecture Decision Record. ADRs have been a critical part of the better orgs I’ve worked in. When someone new joins and asks “why did we build it this way?” the decision log prevents rework and preserves institutional knowledge.
So the expanded definition I’m working with: memory isn’t just memory. It’s memory + diary + decision log. Together, they form the complete subsystem:
Memory stores the what — facts, preferences, processes
Diary stores the path — narrative, thinking process, evolution of ideas
Decision log stores the choices — what was decided, why, based on what, what was rejected
If you build only the first one, you’re missing context that matters. And this matters even more with autonomous agents because the decision log becomes, along with PRs, your best chance to review the work done (reading the full chat would most likely be too much in most cases).
The landscape: how people are doing it today
Claude Code’s built-in memory is, frankly, not great (it used to be terrible). It used to just shove memories into your CLAUDE.md file. Every memory becomes part of your instructions, loaded every session, bloating your context window. No categorization, no retrieval logic, no decay.
Anthropic is now rolling out a new memory system that tracks what the community has been doing for a while, but the implementation is ultimately on you: you create an index table inside your CLAUDE.md with a list of memory files and trigger keywords. When the agent sees a keyword in your prompt, it loads the relevant file. It’s essentially lazy loading, and it’s what I’ve been using. But it’s fragile. I call it “prompt and pray.” You’re hoping the agent notices the keyword, makes the right call about which file to load, and remembers to do it even as the context window fills up. Sometimes it works. Often it doesn’t, similarly to how skill invocation often times fails.
Beyond this, there’s a growing ecosystem of community projects. I’ve spent some time looking into several of them and learned a lot, altho I’d loved to see the diary and decision log of how they got there.
Claude Diary, by Lance Martin from LangChain, implements a /diary and /reflect command pair inspired by the Generative Agents paper. The diary captures session observations; the reflect step finds patterns across entries (two occurrences equals a pattern, three equals a strong one) and updates your CLAUDE.md. It’s manual: you review before changes get applied. Good for PR feedback patterns, git conventions, anti-patterns.
Jesse Vincent’s episodic memory (at fsck.com) takes a different approach: it archives all your Claude Code conversations into a SQLite database with vector search, then exposes an MCP tool so the agent can query its own past sessions. It uses a Haiku subagent to manage context bloat. The key value: it captures institutional knowledge. Not just facts, but reasoning, trade-offs, rejected alternatives.
claude-mem by thedotmack has gotten real traction (24K+ GitHub stars as of February 2026). It hooks into five lifecycle events, stores memories in SQLite plus a Chroma vector database, and uses progressive disclosure (three layers of detail) for roughly 10x token savings compared to loading everything at once.
Rohit’s production architecture is the most systematic approach I’ve found. Three-layer file hierarchy (raw data → atomic facts → evolving summaries), a context-graph combining vector store and knowledge graph, built-in conflict resolution, and maintenance cadences: nightly consolidation, weekly summarization, monthly re-indexing. His mental model (agents as operating systems, with the context window as RAM and persistent storage as the hard drive) is one of the clearest framings I’ve come across.
We’re early enough that everybody is taking a crack at this and trying to build something that works first and foremost for them. Which is great, and that’s what I’m doing too.
My approach: markdown, git, no RAG (for now)
I briefly tried building on top of RAG systems and didn’t like the experience. The moment you add a vector database and embedding pipeline, you lose visibility into what’s actually happening. You can build dashboards and logs, but it’s not the same as opening a markdown file and seeing exactly what your agent knows about you.
So my current hypothesis (and I want to emphasize that word, hypothesis) is that markdown files backed by git are the right foundation for personal agent memory. (For a production application, the answer will be very different.) Here’s why:
They’re human-readable. I can open any file and inspect or edit my memories directly
They’re introspectable. I can see exactly what’s there and what’s not
They live in git. I get version history, diffs, and backup for free
They’re LLM-native. Models are trained on markdown; it’s a natural format
Zero infrastructure. No databases, no services, no maintenance overhead
There’s some validation beyond my gut feeling. Letta ran a benchmark (LoCoMo) comparing different memory implementations, and plain filesystem-based memory achieved 74% accuracy, beating some specialized memory libraries. That’s not perfect, and the remaining 26% gap might matter for some use cases. But for a personal agent? It’s a strong starting point.
Dan Giannone wrote a compelling critique arguing that the whole vector-store-as-memory paradigm is fundamentally flawed: it treats memory as snippet retrieval rather than structured knowledge. He called current implementations “a half-baked first attempt at a genuinely hard problem.” I think he’s right about the diagnosis, even if the alternative isn’t clear yet.
Where I might be wrong
There are counterarguments I take seriously.
Some practitioners argue that better system prompts and well-organized codebases eliminate the need for persistent memory entirely. That’s valid for single-project, single-developer use cases. It breaks down the moment you have multiple projects, evolving preferences, or long-running agents that need to operate across sessions.
Others point to growing context windows (Gemini at 2 million tokens, Claude at 1M as of Opus 4.6) and argue you’ll eventually just load everything. My counter: I don’t think we’ll get past roughly 1 million tokens in practice for most models, and even if we do, inference costs and speed will always favor smaller, targeted context windows. RAG has its own costs, but targeted retrieval should outperform brute-force context loading. This might change once we move past transformer architecture. For now, I’m betting on memory. I’ll acknowledge this is a hypothesis, but it’s where I stand.
In the end this is all text files. It all lives in git. I love when systems are simple, introspectable, easy to understand, easy to maintain. If I can avoid standing up another service, that’s a win.
Self-learning: the reflect pattern (and the risk of becoming dumber)
Memories don’t just appear. They need to be extracted from the flow of work. This is the self-learning piece, where memory and learning become inseparable.
Like others, I’ve implemented a /reflect command that runs at the end of a session, before the context clears (or before compacts if I get there by accident). It scans the conversation looking for specific signals:
Corrections: language like “no, don’t do it that way, do it this other way.” These are the highest-priority memories. Direct user feedback, authoritative, should be applied immediately
Consensus signals: “that was great,” “this worked really well.” The equivalent of a thumbs-up. These confirm existing approaches
New insights: facts, preferences, or patterns that weren’t already in memory
The reflect step then compares what it found against existing memory. Sometimes it discovers something genuinely new. Sometimes it finds that a correction it flagged is already stored, which is actually a more interesting signal. It means the retrieval system failed, not the memory system. The memory was there; it just didn’t get loaded when it was needed.
Anthropic released their own version of this with the /insights command in Claude Code. I tried it. It’s… meh. Not bad, but it didn’t surface much I didn’t already know. The direction is right, though, and it signals they’re paying attention to this problem.
Here’s the catch: this process only works well as long as I stay involved. Out of eight suggestions, there are almost always three that are incorrect. With the long-term goal of autonomous agents in mind, this reflect mechanism needs to improve substantially, and be paired with evals, to make sure the agent doesn’t actually become dumber or more confused over time.
The other hard problem: retrieval
This is where I have the most problems and the least confidence.
Everything I’ve described so far (organizing memory, building diary entries, running reflect loops) is the storage and acquisition side. It works reasonably well. But getting the right memory to the right place at the right time is genuinely hard.
The current approach is what I’ve been calling “prompt and pray.” You have keywords in your CLAUDE.md index table, you hope the agent notices them when you ask a question, and you hope it makes the right call about which files to load. As the context window fills up over a long session, the agent gets worse at these decisions. The instructions sit at the top of the context, so they don’t get completely lost, but a full context makes the right retrieval call harder and harder.
The other problem is granularity. When the agent does load a memory file, it reads the entire thing. It doesn’t treat memory files the way it treats code, finding the relevant function and loading just that fragment. So you either load too much (wasting context) or don’t load at all (losing information).
Here’s how I’m thinking of solving this:
A hook fires on every user prompt submission (Claude Code’s UserPromptSubmit hook). That hook triggers a small, fast subagent, probably running on Haiku for speed and cost (could be Qwen3 on Ollama), whose job is to analyze the prompt’s intent. What is the user trying to do? What context would help? The subagent scans a memory index, identifies relevant fragments (not whole files), and injects them into the main conversation’s context.
Session state matters here. You don’t want to re-surface the same memories every turn. So I’m tracking which memories have already been loaded in a JSON file that resets on session startup or clear, but persists across resume and compact operations.
Then there’s the meta-learning angle: the subagent itself can accumulate its own memory about how to retrieve effectively. It learns which retrievals were useful and which were noise, based on my feedback. We get meta.
This is still mostly theoretical. I have the design, I have pieces implemented, but I need to iterate a lot more. Part of putting this out is hoping to get feedback that validates I’m on the right track, or redirects me.
The cost problem
Here’s something that doesn’t get talked about enough: proper memory retrieval is expensive.
Think about what a full pipeline looks like for every single user prompt (assuming API usage, no subscription plans):
Intent recognition: a small model analyzes what the user wants (~$0.00025 per query with Haiku)
Query expansion: rephrasing and decomposing the query for better retrieval (~$0.00014)
Embedding generation: turning the query into a vector (~$0.000004, basically free)
Re-ranking: scoring retrieved candidates for relevance (~$0.002 with Cohere)
Context assembly: a larger model assembles the relevant context (~$0.014 with Sonnet)
That adds up to about $0.016 per query. At a hundred queries a day (not unusual for someone actively working with an agent) that’s roughly $48 a month. Just for the routing layer. The actual work the agent does is on top of that.
You can strip it down. Haiku for everything, skip re-ranking, keep it minimal: ~$0.001 per query, ~$3 a month. Workable. But at scale, a thousand queries a day or a product with multiple users, the math gets ugly. $6 to $600 a month just for memory retrieval, depending on how sophisticated your pipeline is.
There are mitigation options. A subsidized plan helps for personal use, but even then you burn through your allowed tokens fast. Prompt caching can cut 90% off repeated system prompts. Local models on Apple Silicon via MLX eliminate per-token API costs entirely; you’re paying electricity, not Anthropic. And heuristic pre-filters can catch obvious cases before invoking the full retrieval agent.
But there’s no great solution right now. AI work today is expensive. If you’re on a subsidized plan from a frontier model company, it’s manageable. If you’re trying to build a memory product, not just personal use, multiply these costs by every user. The unit economics don’t work at current pricing without heavy optimization.
This will change. Inference costs keep coming down. But we’re not there yet. For anyone wanting to implement proper memory retrieval today without a subsidized plan, cost is a real barrier, and it’s worth understanding the numbers before committing to an architecture.
Where this is going
Four months in. I have a working system built on markdown files and git, a reflect command that extracts learnings, a priority taxonomy for memory types, and a design for a retrieval subagent that I’m still implementing.
Some of this might be wrong. The markdown-only approach might hit a ceiling that forces me toward RAG. The subagent retrieval might be too slow or too expensive. The cost problems might not improve as fast as I hope.
But I’m increasingly sure about one thing: memory is not optional. It’s not a feature that makes your agent slightly more personalized. It’s the foundation that makes everything else possible: self-learning, research, autonomous operation. Without it, you’re starting from zero every session. With it, the agent gets better over time. That’s the difference between a tool and a collaborator.
This is one piece of the path to autonomous agents. Research, containers, and autonomous operation are coming next.
Also in case it’s helpful, here’s the full deep research plus my notes as the source for this piece and my implementation. Enjoy!
And if any of this resonates, if you’re building something similar or interested in the same problems, I’d love to talk about it. I’d rather have a conversation than a subscriber, so feel free to 📅 find some time to chat.