AI agent memory sounds simple until the agent has to live with its own history.
A chatbot can answer from the current conversation. A search app can retrieve documents. A RAG system can pull a few chunks into a prompt. An agent has a harder job: it must remember user preferences, active goals, unresolved commitments, tool results, project state, prior failures, and the difference between facts it should keep and junk it should forget.
That is why the OpenClaw and Hermes research is useful beyond those two projects. OpenClaw makes memory concrete with inspectable workspace files and retrieval sidecars. Hermes leans toward curated persistent memory, session search, skills, goals, and execution backends. Letta pushes even harder on memory-first architecture. The shared lesson is not that one memory system wins. The lesson is that memory becomes a systems problem as soon as an agent can act across sessions.
🧭 RAG Solves Retrieval, Not Memory
RAG is good at one job: find relevant external context and place it near the model when it answers.
That matters. Agents need documents, previous messages, issues, web pages, transcripts, tickets, code, and tool logs. But retrieval is not the same thing as memory. A vector search result can be semantically similar and still be wrong for the moment. A stale preference can outrank a newer correction. A chunk can contain attacker-controlled instructions. A summary can compress away the exact detail that mattered.
The research bundle describes OpenClaw’s QMD sidecar as combining BM25, vector search, and reranking, plus Active Memory as a bounded recall step before the main answer path. That is a stronger design than “embed everything and hope.” It acknowledges that lexical search, semantic search, reranking, and bounded context all solve different parts of the problem.
Still, retrieval remains a read path. Agent memory also needs a write path, an update policy, a deletion policy, provenance, review, and evaluation.
🧭 Agent Memory Has Several Jobs
A useful agent memory system has to do more than recall notes.
It should preserve durable facts and preferences. It should track open loops, not only past text. It should separate identity, workspace state, task state, tool state, and temporary conversation state. It should help the agent recover from failure without repeating the same broken plan. It should make sensitive facts deletable. It should let a user inspect what the agent believes.
That is why plain files are still attractive. OpenClaw’s Markdown-style memory is easy to diff, back up, edit, and review. A human can see what changed. The downside is also direct: if bad memory is written, it becomes durable state. Memory poisoning is not a theoretical category when the agent reuses stored instructions in later sessions.
Hermes takes a different shape. Its docs, as summarized in the research bundle, emphasize curated bounded memory, session persistence, skills, persistent goals, and multiple backends. That makes it better aligned with long-horizon agents, but it also means governance moves from “what did the user type?” to “what did the agent decide to keep, summarize, and reuse?”
🧭 Search Has Limits Agents Cannot Ignore
Search can fail in quiet ways.
Semantic search retrieves by similarity, not truth. BM25 retrieves by terms, not intent. Rerankers improve ordering, but they do not prove that the retrieved item is safe, current, or complete. Summaries save tokens, but they are lossy. Context windows are finite, so every retrieval choice is also an exclusion choice.
For agents, the common failure modes are predictable:
- The agent retrieves an old preference after the user changed their mind.
- A search result is topically similar but tied to the wrong project.
- A malicious web page or email gets stored as useful context.
- A summary drops a deadline, exception, or negation.
- A tool log is treated as user intent.
- The agent remembers a failed workaround as a recommended procedure.
The fix is not simply “use a bigger context window.” Bigger windows increase capacity, but they do not create lifecycle policy. The hard part is deciding what deserves to become durable state.
🧭 Memory Needs a Lifecycle
Production memory needs lifecycle design.
First, classify memory before storing it. A user preference is not the same as a project fact. A project fact is not the same as a tool result. A tool result is not the same as a learned skill. A retrieved web page is not the same as a trusted source.
Second, attach provenance. The agent should know where a fact came from, when it was written, whether the user approved it, and whether it was inferred. Inferred memory is useful, but it should not have the same authority as explicit user instruction.
Third, support expiry and correction. Some facts should age out. Some should be replaced. Some should require review before they are reused in high-impact workflows.
Fourth, make deletion real. If memory stores private data, PII, credentials, client names, medical facts, or financial details, the system needs a way to remove them from active use. “The model probably will not mention it” is not a deletion strategy.
🧭 Skills Are Procedural Memory
The research bundle calls out an important point: skills are memory too.
A skill is procedural memory. It tells the agent how to perform a class of work next time. Skill systems can include instructions, scripts, references, and assets. They are powerful because they let the agent page in specialized knowledge only when needed. Progressive disclosure keeps the working context smaller: advertise the skill name and description first, then load detailed instructions and resources when the task matches.
That is good context engineering. It is also a supply-chain boundary.
If a skill can read files, run scripts, call APIs, or install dependencies, it is not passive documentation. It is privileged software. Teams should review skills the way they review plugins, internal runbooks, and automation scripts.
🧭 Evaluate Memory Over Time
Most demos evaluate memory too early.
A one-session recall demo proves little. The real question is what happens after days or weeks of contradictory updates, partial tasks, noisy documents, failed tool calls, and changing user preferences.
A practical memory evaluation should include:
- Recall correctness across multiple sessions.
- Stale-memory incidence after user corrections.
- Poisoning resistance from untrusted documents.
- Latency and token cost added by retrieval.
- Whether the agent can explain why it remembered something.
- Whether memory edits and deletions affect later behavior.
- Whether procedural skills improve success without adding unsafe authority.
Letta’s memory and context benchmarks, Hermes’s session and trajectory tooling, and OpenClaw’s inspectable memory model all point in the same direction: memory quality should be measured as a long-running system property, not as a single chat feature.
🧭 A Better Mental Model
Do not ask whether an agent “has memory.” Ask what kind of state it owns.
There is working context for the current turn. There is retrieval context from external sources. There is durable user memory. There is project memory. There is tool state. There is procedural memory in skills. There are traces and evaluations. Each layer has a different trust level and lifecycle.
The safest design is not the most magical one. It is the design where memory is inspectable, bounded, editable, attributable, and tested against real tasks.
RAG is still useful. Search is still useful. But for AI agents, memory begins where retrieval ends.
🧭 Sources and Further Reading
- OpenClaw official site – self-hosted assistant and gateway positioning.
- OpenClaw documentation – workspace memory, QMD, Active Memory, gateway, and security references surfaced in the research bundle.
- Hermes Agent documentation – memory, skills, profiles, gateway, and backend architecture.
- Hermes Agent GitHub – project implementation and release context.
- Letta documentation – memory-first agent architecture, memory blocks, and context engineering.
- MemGPT paper – OS-inspired virtual context management for LLM agents.
- Anthropic: Building Effective Agents – workflow and agent design patterns.
Recommended Books & Tools
If you are building agent memory, the best supporting material is systems engineering, information retrieval, and production AI evaluation.
Books for Agent Memory Builders
- Designing Data-Intensive Applications – A practical foundation for thinking about state, logs, durability, consistency, and data lifecycle.
- AI Engineering: Building Applications with Foundation Models – Useful for moving from model demos to production AI application design.
- Introduction to Information Retrieval – Search quality matters when RAG and memory systems depend on retrieval.
Development Tools
- Samsung T7 Shield Portable SSD – Useful for encrypted local snapshots, evaluation datasets, and reproducible memory experiments.
- YubiKey 5C NFC – A hardware security key is a practical baseline when agents touch developer accounts and private data.
Read the systems material first. Agent memory fails at state boundaries, provenance, and evaluation before it fails at embedding math.