The setup
OpenClaw agents keep long-lived workspaces. The default way to give the agent its memory is to inject whole workspace files into context, which costs about 15,000 tokens every turn and still misses things. A colleague built HGD, a memory engine that replaces file injection with compact retrieval: memories split into semantic facts, episodes, incidents, and decisions, stored in SQLite with full-text search, hashed sparse vectors, and a knowledge graph, assembled per query into a small memory packet.
My job was to answer a different question: does it actually work? Engines like this demo well and then fail on the queries that matter. So I built the evaluation and ran it against every version.
What I measured
I wrote a 20-query benchmark over a real 78-file, 129K-token workspace, covering 8 categories: personal facts, infrastructure, recent events, PR history, root causes, decisions, multi-hop questions, and negative queries (asking about things that do not exist, where the right answer is "nothing found"). Each query has a predefined expected answer and gets scored correct, partial, or missed.
Recall accuracy by engine version
20-query benchmark, same corpus throughout
View as table
| Version | Recall |
|---|---|
| v1, lexical search only | 40% |
| v2, episodes + hashed vectors | 60% |
| v4, recall flags, incidents, archive fallback | 70% |
| v5, default concept aliases | 55% |
| v5, empty concepts | 80% |
| v5 + subject-coverage fix | 85% |
Context tokens per turn
78-93%
lower than full-file injection
Average packet size
~3,300
tokens, vs ~15,000 before
Retrieval cost
$0.001
per query, LLM routing included
The ablation that surprised me
The engine supports real transformer embeddings (bge-small, MiniLM) as a drop-in upgrade over its default feature-hashed sparse vectors. Everyone's instinct, including mine, was that real embeddings would win. They did not. All three scored exactly 70% on the benchmark, and the transformer embeddings cost 30x more import time and 20x more query latency for it. On a corpus like this, retrieval quality was bottlenecked by how memories get split and typed, not by embedding quality.
Cross-encoder reranking was worse than useless: it dropped accuracy from 80% to 40%, because a reranker trained on passages scores single-sentence fact chunks badly. And the version 5 regression in the chart taught the same lesson from another angle: the shipped default concept aliases actively hurt (55% vs 80% with empty concepts). Every "obvious upgrade" needed the benchmark to call it.
The fix I contributed
At 80%, the remaining failures were confident wrong answers, mostly on negative queries: ask about something that does not exist and the engine would return its closest match anyway. I wrote a subject-coverage confidence check: extract the query's subjects, verify the retrieved evidence actually covers them, and detect does-X-exist style questions so the engine can answer "nothing found" with confidence instead of guessing. That closed the gap from 80% to 85%, which is the measured ceiling for this corpus, since the remaining misses are data gaps rather than retrieval failures.
Numbers to keep in context
- 20 queries, hand-scored, one private corpus. This is a rigorous small benchmark, not a public leaderboard.
- The 85% is corpus-specific. I would not quote it as a general accuracy claim, and neither should you.
- The engine was a colleague's build. The benchmark, the ablations, and the confidence fix are mine.
What I took from it
A small benchmark you actually read beats a big one you dashboard.
Retrieval systems fail quietly, and the failures cluster exactly where demos never look: negative queries, multi-hop questions, stale defaults. A small benchmark you actually read beats a big one you dashboard. And measured negative results, like reranking hurting or embeddings not mattering, saved this project more tokens and latency than any feature added.