Intelligence Doesn't Cure Amnesia

At one point I had a memory system that could find almost anything. Tags, embeddings, full-text search, a knowledge graph, recall in milliseconds. My agents ignored it.

Not on purpose. Every prompt arrived with a neat block of "relevant memories" on top, and after a few turns the model treated it the way you treat a cookie banner: always there, never read.

That was when I stopped thinking of memory as storage.

Sprint versus marathon

Most of the AI conversation right now is about intelligence. Which model is smarter, which one reasons better, which one tops this week's benchmark.

For short tasks that is the right question. Write a function, summarise a document, fix a bug in one file. A smart model does it, and the session ends before anything is forgotten.

Real projects are not short tasks. They run for weeks and months. Decisions pile up, approaches die, constraints appear, and somebody has to remember why the code looks the way it does.

A genius who wakes up with amnesia every couple of hours cannot build anything more complex than a calculator can. The smarter the model, the faster it confidently rebuilds the thing you already rejected last Tuesday.

So the question I ended up working on was not "how do I make the agent smarter?". It was "how do I make a long piece of work survive an agent that keeps forgetting?".

Writing is easy. Recall is the product.

Anyone can write a note. Any database can store one. That part is solved.

The hard part is everything after:

• What is worth keeping at all?
• At what level: this task, this project, or everything I do?
• When does it go stale?
• What should trigger it coming back?
• Does it actually change what the agent does next?

A record that nobody ever recalls at the right moment is not memory. It is an archive. You can have a very large, very well-indexed archive and still have an agent with amnesia.

What I tried, roughly in order

A file protocol. The first memory was not software at all. One index file as the entry point, plus append-only logs of decisions and errors. Every session started by reading them. It worked better than you would expect, and it depended entirely on the agent actually reading them.

Tags. Then I automated the "remember to look" part. The incoming message was split into tags, the memory store was searched by those tags, and the titles of matching records were injected into the agent's context. Now the agent did not have to know a record existed; the record came to it.

Semantic search. Exact tags miss things, so the next step was obvious: embeddings, so that related records surface even when the words do not match. Retrieval got much better.

Letting a model write the memories. At some point I also let a local model read raw chat and turn it into memory entries automatically. It chopped conversations into 397 fragmented, unusable entries. I switched it off. Memory writes became explicit again: an agent decides something is worth keeping and writes it deliberately, and a quality gate rejects duplicates and noise.

Each of these steps improved retrieval. None of them, on its own, improved use.

The wallpaper effect

Here is the part I did not expect.

The better the retrieval got, and the more reliably memories were injected, the less they mattered. If something appears in every prompt, the model learns that it is part of the furniture. It is the same thing that happens to people with banners on websites. Constant presence turns a signal into background.

So more memory in the context was not the fix. In some cases it was the problem.

Cleaner, not bigger

That flipped the goal. Instead of "put as much relevant memory in front of the agent as possible", it became "keep the context clean, and put the right signal in front of it at the moment it matters".

This is not only about attention. Context is also a budget. When I moved agents from vendor tooling to direct model calls, where my own platform decides exactly what enters the context window, input tokens per run dropped by 89 to 92 percent, and the agent still received the project's current direction and the task in hand. Most of what used to sit in the window was never doing any work.

Put the signal where the work is

The other half of the answer was not about memory at all. It was about slicing the work.

I stopped trying to fit a project into the agent's head. Instead the project is cut into small units of focus: a task, its goal, the hypotheses still alive, the evidence so far, the constraints, the next step. The agent works inside one of those units.

That changes where warnings live. A constraint that matters for this task is written on the task card, not repeated in every turn of every conversation. It is much harder to ignore something that belongs to the work you are doing than something that floats around it.

Over time that board of focus units became the agent's working surface: what is in progress, what is open, which hypotheses are dead, what was already checked, what comes next. The agent does not need to hold the project in its context. It needs to see the right card.

Memory still has a role, but a narrower one: it comes back at the start of a session, when an agent picks up someone else's work, and at the checkpoints where a decision is being made. Not as a permanent layer of wallpaper.

Memory is not truth

One more distinction took me too long to make.

Memory helps you remember. The current state of the project tells you where you are. Evidence tells you why you believe something. These are three different things, and mixing them causes real damage.

A memory that says "we use approach X" is useful until the project quietly moved to approach Y last week. If memory is allowed to override the current state, an agent will happily restore something you already replaced, and it will do it with full confidence, because it remembered.

So fresh state wins. Memory is background and history. It is never allowed to overrule what the project says is true right now.

A Monday checklist

If you run agents on long work, here is what I would check first:

Measure what you inject. How many tokens of memory, instructions and context go into each run?

Check what changed a decision. Of everything injected, which items actually changed what the agent did? Delete the rest.

Record dead ends. A rejected approach is one of the most valuable things to remember, and the first thing everyone forgets to write down.

Mark freshness. Every memory should carry some sense of when it was true.

Move warnings onto the work. If a constraint matters for one task, attach it to that task, not to every prompt.

Let current state win. When memory and the project disagree, the project is right.

The question to ask instead

The industry keeps asking how smart the next model will be. It is a fair question, and the answer keeps getting better.

But for long work, a smarter model with amnesia is still a model with amnesia. The more useful question is: what can this agent recall at the moment it matters, and what have you kept out of its way?

---

*Ali Agzamov, BrainOps*