Your agent's memory is probably mostly garbage

Agent memory doesn't fail the way you'd expect. It doesn't crash. It doesn't throw. It rots — slowly, quietly, and in a way that looks fine right up until an agent says something that makes a customer wonder whether anyone's home.

A public audit of one popular memory layer found that after a month of normal use, the large majority of stored entries were effectively garbage: duplicates, contradictions sitting side by side, and stale facts that still outranked the truth. That's not one vendor being careless. It's the default outcome of how most agent memory is built. Let's talk about why — and what it takes to fix it.

Append-only memory is a hoarding problem

The default design for agent memory is: extract facts from a conversation, embed them, store them. Retrieve by similarity when you need them. Simple, and it demos beautifully.

The trouble is that nothing ever leaves. Every design decision points toward keep more:

• A user says "I'm vegetarian." Later they say "actually I eat chicken now." Both get stored. Both come back on retrieval. The agent picks one — often the wrong one.
• The same preference gets restated five times across five sessions. Now there are five nearly identical memories, all competing for the same retrieval slot, crowding out everything else.
• A fact that was true in January is still sitting there in July with full confidence, because nothing ever told it to fade.

Multiply that across thousands of interactions and you don't have a memory. You have a landfill with a search bar.

Why contradictions survive

Here's the part that surprises people: similarity search actively hides contradictions.

"I am vegetarian" and "I am not vegetarian" are nearly identical as text. To an embedding model they're neighbors — high cosine similarity, because they're about the same thing. So a naive system looks at the pair and concludes they reinforce each other. The negation, the single most important word in the sentence, is invisible to the geometry.

We learned this one the hard way in our own system, and wrote about the fix in detail. The short version: catching "X" versus "not X" needs something that understands entailment, not just distance. Distance will tell you two beliefs are about the same thing. It will not tell you they disagree. If your memory only knows distance, contradictions don't get resolved — they get stored twice and averaged into confusion.

The other failure: confidence that never moves

The subtler rot is confidence that doesn't respond to reality. In a lot of systems every memory is equally, permanently true. The preference you set once and abandoned carries the same weight as the one you've confirmed a dozen times. The fact from six months ago sits at the same confidence as the one from this morning.

When everything is equally true, retrieval has no way to prefer what's actually current. And "confidently wrong" is a far worse failure mode for an agent than "I'm not sure" — because your software acts on it without hesitating.

What self-cleaning memory does instead

The fix isn't a bigger vector index. It's giving memory the thing human memory has and most agent memory lacks: the ability to forget on purpose.

In Hakuya, that's a few mechanisms working together:

• Confidence decays. Beliefs that aren't reinforced lose confidence over time. Stale facts fade instead of lurking at full strength, so fresh information naturally wins retrieval.
• Reinforcement and contradiction move the needle. Confirm a belief and it strengthens; contradict it and it weakens. Confidence is a running consequence of evidence, updated with log-odds math — not a number you set once and forget.
• Competition-aware decay. Redundant memories suppress each other instead of all fighting for the same slot. Five restatements of one preference collapse toward one strong belief rather than five weak duplicates.
• Contradiction detection that survives negation. Conflicting beliefs get surfaced and resolved — demoted, archived as superseded, or kept as genuine contextual variants — instead of quietly coexisting.

The result is memory that gets cleaner the more it's used, not dirtier. The signal concentrates; the noise fades. You don't run a cleanup job. The system maintains itself.

The tell

Here's a quick diagnostic for your own stack. Ask your agent something it should know a user changed their mind about. If it confidently gives you the old answer — or worse, gives a different answer each time you ask — your memory is rotting. It's not retrieving the truth; it's retrieving whatever happens to be nearest in vector space, and the truth got outvoted.

Memory that can't forget isn't memory. It's an archive that gets less trustworthy every day it stays up.

If you want to see what the alternative looks like, Hakuya is open source — belief dynamics, competition-aware decay, and all. And the audit trail means you can always go back and see exactly why a belief faded, strengthened, or got superseded. Because self-cleaning is only trustworthy if you can prove what got cleaned.