What to Remember, What to Forget

Here is a failure every chatbot with memory eventually hits. This one took seventeen months.

The obvious reaction is "the bot forgot." It did not. The fact was stored correctly and was retrievable the entire time.

Here is what the model actually saw on that last turn:

Every retrieved chunk was accurate. But the system missed the most useful fact of all, which is that the user is vegetarian. They said so in January 2025. The search looks for messages that sound like the question. Old messages about dinner places sounded much more like it than "I'm vegetarian," so they won and the important fact was left out. The user stated it directly once in seventeen months and never repeated it. Why would they? They expected the system to remember.

The setup that produces this

Almost every first build of a RAG-based chatbot looks like this:

1. Save every message to a vector database
2. Embed the incoming query
3. Retrieve top-k by cosine similarity
4. Paste into the prompt

This setup demos beautifully but degrades over time. The main issue is that a chat log is not a document. A document is stable because once it is indexed it stays the same forever and its parts connect to each other. A conversation is the opposite. The user changes their mind, corrects you, states something important exactly once and says "ok thanks" four hundred times.

Similarity search has no opinion about any of that. It ranks by embedding distance and nothing else. It cannot tell that "I'm vegetarian" will stay true for years while "I'm stuck in traffic" stops being true within the hour. To the search they are both just text.

The underlying mistake is simple. Episodic, semantic and procedural memories all end up in one vector store with identical retrieval logic.

That is the bug.

Why a bigger context window doesn't save you

You may wonder why we can't just paste in everything and let a long-context model sort it out. Either the history is too big for the context window or if it fits then you pay full price on every single turn for a history that is 99% irrelevant.

So something has to select. The question is not whether to select but what the selection runs on. Everything below is about that.

Memory is retrieval plus a write path

Retrieval is a library. Books on shelves. You search, read, leave. You never change the books.

Memory is a notebook. You write in it. You cross things out. You add margin notes. It changes every day.

Memory does not replace retrieval because it still needs an index to find anything. Mem01 splits its write path into two phases called extraction and update. Even the update phase starts with a vector search for similar memories. The reading half never went away.

So the interesting question is not which retrieval algorithm you use. The real question is what happens between the user's message and the database write.

Most articles diagram the read path. This one goes the other direction.

The four stages are extract, classify, reconcile and forget. The diagram shows the first three. Forget is different because it runs later on what is already stored. Note that one outcome is "skip." Doing nothing is a valid result and it is the one most systems never implement.

Stage 1: Extract

Raw turns make terrible memories. "btw I'm vegetarian, been about six years now" is a message. What you want to store is the claim inside it.

INPUT:  "btw I'm vegetarian, been about six years now"

OUTPUT: [{
  "content": "User is vegetarian",
  "type": "semantic",
  "scope": "constraint",
  "beneficiary": "self",
  "confidence": 0.95,
  "importance": 9
}]

Compare with the March message:

INPUT:  "booked the Main Street steakhouse for the
         team dinner, everyone was happy with it"

OUTPUT: [{
  "content": "User organized a team dinner at
              Main Street steakhouse; team liked it",
  "type": "episodic",
  "scope": "event",
  "beneficiary": "team",
  "confidence": 0.9,
  "importance": 3
}]

Look at what beneficiary captures. The user booked that restaurant for other people. A flat chunk of the raw sentence loses that completely. That is why it reached the model looking like a personal dining preference.

Extraction is where you get to keep the distinction. Nowhere downstream can recover it.

What gets stored

Here is what one memory looks like on disk.

{
  "id": "mem_8f21",
  "user_id": "u_1042",
  "type": "semantic",
  "scope": "constraint",
  "beneficiary": "self",
  "content": "User is vegetarian",
  "source_session": "s_003",
  "created_at": "2025-01-14T10:22:00Z",
  "valid_from": "2019-01-01",
  "valid_until": null,
  "importance": 9,
  "confidence": 0.95,
  "supersedes": null,
  "last_accessed": "2026-06-02",
  "access_count": 31
}

Every field prevents a specific bug.

valid_until retires a fact without deleting it so the history is kept. supersedes gives an audit trail when a preference changes. confidence stops a hedged remark being treated like a declaration. beneficiary separates what the user did for themselves from what they did for others so a team booking never becomes a personal preference. valid_from predates created_at here because the user said "six years." The fact is older than the conversation that surfaced it.

importance needs a caveat. That brings us to the next stage.

Stage 2: Classify and why "usefulness" is a trap

The instinct is to score memories by usefulness. It sounds right and it does not work.

Take the record above. How useful is "User is vegetarian"?

Unanswerable. It is decisive for "suggest somewhere for dinner tonight for me." It is worthless for "fix my Python function." Usefulness is not a property of a memory. It is a relationship between the memory and a question you have not seen yet.

This is where the importance caveat comes in. Importance is not usefulness. It measures how costly it is to get a fact wrong. Forgetting that a user is vegetarian is far worse than forgetting where the team ate.

What you can determine at write time is what kind of thing it is. The CoALA framework2 from Princeton names four standard types. They are working memory (the active window), episodic (records of past interactions), semantic (factual knowledge) and procedural (rules and skills).

Route them separately:

Read the right-hand column. Four types have four retrieval methods and only one of them is similarity search.

That table is the whole argument. Go back to the context the model saw. "User is vegetarian" should never have had to compete for a place in it at all. It is a constraint. It should be injected on every turn without conditions. It should never be ranked and never depend on whether it happens to sound like the query.

Within semantic memory some facts are constraints. A constraint is not looked up. It is loaded on every turn because a user rarely has more than a handful.

Constraints don't compete for top-5 slots. They filter the results.

Procedural memory makes the point even harder. Output preferences should not be retrieved at all. They belong in the system prompt on every single turn.

Stage 3: Reconcile

Now the harder case. New facts sometimes touch old ones.

Three questions on every incoming fact:

1. Do we already know this? → skip
2. Does this replace something we know? → update and set valid_until on the old record
3. Is this genuinely new? → add

For the March message the answer is add. It is a new event. Because the beneficiary was the team it does not touch the user's dietary constraint at all. A correct system stores it and leaves the constraint alone.

Compare that with a real conflict such as "honestly I've started eating meat again". That is a direct supersede. Set valid_until on the old record, write a new one and link them through supersedes.

You will not classify every case correctly. You should still try because the alternative is storing everything as equal chunks and letting cosine similarity decide. That is exactly how the steakhouse suggestion happened.

This problem has a name in the literature now. A 2026 benchmark paper called From Recall to Forgetting3 puts it plainly by saying that real interaction is non-stationary because user information gets updated, corrected or withdrawn over time. So long-term memory needs not just recall but correct handling of memory mutation. The same paper notes that even LongMemEval4 limits knowledge updates to at most two sessions before testing.

This means the benchmarks barely test it. Your users will hit it in week two.

Stage 4: Forget

Not everything deserves permanence.

Decay. "Lives in London" is stable for years. "Stressed about Friday's deadline" is dead by Monday. Assign a lifetime at write time based on type.

Decay does not mean deleting. Take "I'm stuck in traffic and running late." Store it as an episode with a short valid_until.

{
  "content": "User was stuck in traffic and running late",
  "type": "episodic",
  "created_at": "2026-05-12T18:40:00Z",
  "valid_until": "2026-05-12T19:40:00Z"
}

If the user later asks "when was I stuck in traffic?" the event is still there with its date. If they ask "am I running late?" the next day the bot no longer treats it as true. The fact stops being current. It does not stop existing.

Gating. The newest idea is deciding whether to store at all. PerMem-Bench5 proposes session-level storage gating. It skips memory operations entirely for transient sessions. The paper also reports that perfect gating gives real retention gains but doing it accurately is still an open problem.

It is worth stating clearly that knowing what to ignore is still unsolved research. Storing everything is not a safe default. It is an unmade decision. In our example it is what filled the store with thousands of competing chunks in the first place.

Deletion. Under GDPR6 a user can demand that one specific fact be erased. You cannot comply without wiping the profile if memories live inside summary blobs. Atomic records with IDs are not just cleaner. They are the only version that is legally operable.

Where it lands

Here is the same user with the same seventeen months of history and this write path in place:

Nothing dramatic happened. The constraint was loaded without competing for a place. The team dinner was set aside because it was booked for the team. The meal the user actually enjoyed came through instead.

That is the entire difference between the two transcripts and it lives in the write path.

When not to build this

Memory adds cost, complexity and privacy liability. Skip it in these cases:

• Self-contained queries
• Users who have not asked for personalization
• Regulated environments where retention is a risk
• Tasks that are short-lived by nature

Be honest about the baseline too. Pasting it all in often just works when the history fits in a window. Zep pointed out7 that on LoCoMo8 Mem0's own published results showed a plain full-context baseline beating their memory system (roughly 73% against 68%) and argued that if raw text beats the specialized system then the benchmark isn't stressing memory hard enough.

So the case for a memory layer is not primarily accuracy. It is cost and scale. Mem0 reports1 using about 7,000 tokens per conversation where the full-context approach uses about 26,000. It also reports 91% lower p95 latency.

That is roughly a quarter of the tokens. It also still works at 1.2M tokens of history where the full-context approach becomes slow and expensive.

One warning if you go looking for numbers. Zep first reported 84% on LoCoMo. Mem0's CTO replied9 that the real figure was 58.44% because Zep had counted adversarial questions that the benchmark excludes. Zep then corrected7 its score to 75.14% and argued that Mem0 had set Zep up wrongly in its own tests. Two vendors ran one benchmark and got wildly different results. Always write "Mem0 reports" and never "studies show."

The real bar

A system can recall every fact correctly and still fail to personalize. The PersonaMem10 benchmark exists for exactly this gap. A user who keeps asking for shorter and less formal writing without ever saying "this is my preferred style" should eventually get that style by default.

That is the difference between a chatbot that remembers and one that understands. The first is a retrieval problem. The second is a write-path problem.

References

1. Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. 2025.
2. Sumers, Yao, Narasimhan and Griffiths. Cognitive Architectures for Language Agents (CoALA). 2023.
3. Uddin et al. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. 2026.
4. Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. 2024.
5. In et al. Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents (PerMem-Bench). 2026.
6. GDPR Article 17: Right to erasure.
7. Zep. Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory? 2025.
8. Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo). 2024.
9. Mem0. Revisiting Zep's 84% LoCoMo Claim. GitHub issue, 2025.
10. Jiang et al. PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. 2025.