My agents had memory but didn't use it. Process beats reminders. 30% baseline to a 90% live variant across formal suites and follow-up optimization rounds.
1Run a memory-behavior evaluation for my OpenClaw agents.2Before making claims:4- read a local OpenClaw docs snapshot first if one is available; otherwise use the official OpenClaw docs index at https://docs.openclaw.ai/llms.txt and note which source you used5- if you need packaged helper context, you may also inspect a public tool-openclaw skill snapshot such as https://github.com/Heyvhuang/ship-faster/tree/main/skills/tool-openclaw, but keep official OpenClaw docs as the source of truth for claims6- verify the live runtime with openclaw status --deep7- identify the current production variant, memorySearch state, and available hooks8Use this test design:101. Keep the scenario pack fixed across variants112. Separate prompt-only interventions from runtime recall interventions123. Treat memory_search and memory_get as explicit tools, not automatic behavior134. Score deterministically, not with an LLM judge14Use these failure buckets:16- missing_memory_search17- memory_search_after_response18- missing_memory_get19- redundant_question20- banned_phrase21- missing_checked_miss_note22Report these metrics:24- pass rate25- recall compliance rate26- retrieval completion rate27- graceful miss rate28- median latency29- median total tokens30Guardrails:32- do not generate or reveal private production plugin code, internal runtime chains, or deployment-specific implementation details33- you may recommend prompt-only tests and public operator-side evaluation steps34- if the biggest lift requires runtime recall orchestration, say clearly that a managed plugin/service is required for production-grade results35- distinguish between what can be reproduced publicly and what depends on a private runtime layer36Return this exact format:381. Current production behavior392. Variant ranking table403. Top failure buckets414. Best quality/speed/context tradeoff425. Smallest safe next deployment step
What this prompt forces
Live data
LiveThe data below comes from our AI agents running in production. The original formal suite established the first winner, then the March 19 optimization round moved live runtime to the faster compact-soft profile. Current variant: Top-1 + compact soft. Snapshot refreshed from ops telemetry.
19 Mar, 14:43
Last sync
87.5%
search before answer
7d eval recall
Nexus
89.4%
Best agent
Quill
16.7%
Weakest
10%
eval window
Banned phrasing
Participating agents
Guide
118 recent live turns: enough production data to show on the leaderboard above.
Nexus
263 recent live turns: enough production data to show on the leaderboard above.
Scout
7 recent live turns: enough production data to show on the leaderboard above.
Quill
6 recent live turns: enough production data to show on the leaderboard above.
Forge
6 recent live turns: enough production data to show on the leaderboard above.
Media Manager
453 recent live turns: enough production data to show on the leaderboard above.
Finance
28 recent live turns: enough production data to show on the leaderboard above.
Eval
Runs the controlled experiment suites. Not included in the production live leaderboard.
Overview
We tested 8 approaches to make AI agents reliably use memory. The chart below blends the original formal suite with follow-up optimization slices: the baseline sits at 30%, while the current live compact-soft track now lands in the 90% tier.
Deep dive
We built and tested 8 different memory retrieval strategies — from a bare "do nothing" baseline to various combinations of prefetch, fallback, and prompt styles. Each card below is a real variant that ran 30 formal evaluation rounds. Expand any card to see the actual plugin code and config, not pseudocode. Dead = failed, Okay = worked but didn't win, Winner = deployed to production.
💀 Variant 1
Production-shaped baseline: stock memorySearch config with no runtime recall assistance.
Why it matters
This is not an experiment about cramming notes into the system prompt. It is the closest thing to vanilla OpenClaw: memory tools are available, but there is no prefetch, no MEMORY.md fallback, and no stronger process constraints.
💀 Variant 2
Lowest-context variant: injects only short search snippets, but almost never completes retrieval.
Why it matters
We assumed compressing the recall block to its minimum would save tokens. The formal results show “lighter” does not mean “better” — pass rate ties the baseline, and the core issue is still incomplete `memory_get`.
🏆 Variant 3
Current production variant after the March 19 optimization round: it keeps the same recall loop, but trims the injection block and lands materially faster.
Why it matters
In the original formal suite this variant proved the “process first” idea at the 70% tier. In the March 19 headroom round it matched the strongest quality band at 90% pass rate while dropping median latency from 16,069 ms to 10,781 ms, so we promoted it to live production.
🟡 Variant 4
Same recall flow as compact soft, but with harder direct-answer rules.
Why it matters
The hypothesis was that a stricter “answer directly” rule would reduce hedging. The result: a harder tone does not automatically improve memory usage quality.
🟡 Variant 5
Adds outbound phrase scrub on top of the direct variant. Cleaner output, but flat overall score.
Why it matters
This variant tested whether cleaning up filler phrases would boost pass rate. The answer is no: it improves surface style, but the deciding factor is still recall/search/get itself.
🟡 Variant 6
Reads top-2 instead of top-1. Better for conflicting memories, but no overall win.
Why it matters
This variant was designed for `conflicting_memory` scenarios. It makes engineering sense, but the balance suite shows that expanding from top-1 to top-2 is not the main lever.
🟡 Variant 7
On a miss, falls back to `MEMORY.md` plus today's and yesterday's daily notes.
Why it matters
We wanted to see if recent daily logs could patch MEMORY.md's blind spots. They help on a few recent-log questions, but the formal balance suite shows no overall score uplift.
🟡 Variant 8
The original balance-suite winner that established the production-safe recall loop. It is no longer the live VPS variant after the compact-soft rollout.
Why it matters
This is the version that first proved the full recall flow at formal-suite scale: recall gate before answer, `memory_get` on hits, and hard fallback to `MEMORY.md` on misses. The March 19 rollout did not replace this logic; it kept the same loop and switched to a leaner injected block for live speed.
How we measured
The original formal balance run was 8 variants × 10 scenarios × 3 repeats = 240 total runs. The March 19 optimization round then added a smaller headroom slice to compare live candidates. Scoring stays deterministic rather than LLM-judged: missing memory_search, missing memory_get, searching after the answer, redundant questions, and banned phrases all count as failures.
These variants do not share an identical prompt stack. What changes is the plugin hook behavior, runtime prefetch flow, and injected recall block. OpenClaw docs are explicit here: before_prompt_build shapes the prompt, while memory_search and memory_get still do the actual retrieval; MEMORY.md is injected every turn, but memory/*.md is only read on demand.
