Most systems described as having memory are replaying a transcript. That works until the conversation outgrows the window, at which point the earliest and often most important turns are the first to be dropped. A memory layer is the thing that stops decisions evaporating when the buffer fills.
Three kinds of state, commonly confused
- Working state — the current conversation. Ephemeral, bounded by the window, safe to lose when the task ends.
- Episodic memory — what happened: a summary of an interaction, retrievable later. Useful, and the easiest to let grow into noise.
- Semantic memory — durable facts and decisions: this customer is on the enterprise plan, we chose Postgres over DynamoDB and here is why. This is the layer that changes behaviour.
The test for what belongs in memory
Ask whether the fact should still influence behaviour in six months. "The user said hello" fails. "The user's deployment target is EU-only for data-residency reasons" passes, and if it is not durable, you will violate it the first time the transcript is trimmed.
Write memories as decisions, not as transcript
"We are not using Kafka; the volume does not justify the operational cost, revisit above 10k events/sec" is retrievable, comparable and falsifiable. A stored chat log containing that sentence somewhere is none of those. Structure at write time is what makes retrieval work later.
What a memory record needs
- A claim — one fact or decision, stated plainly.
- Provenance — who or what established it, and when. Without this you cannot resolve contradictions.
- Scope — who or what it applies to: this user, this project, this organisation.
- A status — current, or superseded by a later record. Never edit in place.
- A trigger for review — the condition under which it should be reconsidered.
The contradiction problem
Memory that only appends eventually holds two incompatible claims, and retrieval will happily return both. Some systems resolve this at read time by preferring the most recent; that is wrong as often as it is right, because recency is not authority. Superseding explicitly — marking the old record and pointing at the new one — keeps the history readable and the current state unambiguous.
Retrieval, not injection
A memory layer that dumps everything it knows into every prompt has recreated the context-dilution problem it was meant to solve. Memories are retrieved like any other context: scoped, ranked, and limited. If a memory is not relevant to the current task, its correct contribution is nothing.
Where this overlaps with documentation
A durable record of a decision, its context and its consequences is an architecture decision record. The difference is the reader: an ADR is written for humans and happens to be machine-readable, while a memory record is written for retrieval and happens to be human-readable. Teams that already write ADRs have most of a semantic memory layer and usually have not indexed it.
Frequently Asked Questions
Can I just use a longer context window?
That extends working state, not memory. Memory is about what survives after the task ends and is retrievable by a different session weeks later.
Should memories be per-user or per-organisation?
Both, with explicit scope on each record. Leaking one user's memory into another user's context is a privacy incident, so scope has to be a first-class field and enforced at retrieval.
How do I stop memory from growing forever?
Expiry conditions and supersession, not deletion by age. A three-year-old decision that still holds is more valuable than last week's chatter.
References
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.