Skip to content

Independent evaluations

More hard coding issues solved.

In MemDev-Bench, Claude-Mem gave the coding agent a searchable history of earlier work. On 133 difficult issues drawn from SWE-bench Verified, that agent solved more with repository history than it did with memory off.

MemDev-Bench · Difficult tasks
+7.3 pts

in issue resolution

  • Without memory35.3%
  • With Claude-Mem42.6%

Share of 133 selected tasks resolved, averaged over three runs per task. Repository-history condition.

Published results

Where researchers put our memory to work.

These tests ran Claude-Mem on coding issues, session handoffs and decisions that disappeared from the code. Open each method to see the setup and the comparison.

42.6%Issues resolved with repository history
MemDev-Bench · Independent preprint

The strongest result on difficult issues with repository history.

Chen and colleagues ran five memory systems through the same coding harness. With a history of earlier repository work, Claude-Mem led the five on 133 selected difficult SWE-bench Verified tasks. The no-memory agent resolved 35.3%. Claude-Mem also led with relevant-only history at 41.6%.

Test conditions and comparison

The researchers pinned Claude-Mem v12.1.1 and used Claude Code with DeepSeek-V4-Flash. They froze each history and ran each target three times. On the difficult subset, Claude-Mem resolved 40.1% with noisy history, where Supermemory reached 40.6%. On 50 selected easier tasks, repository history gave Claude-Mem 96.7% versus 94.0% without memory. The subsets were selected using separate baseline runs.

52.00%Overall coding tasks resolved
Agent Memory Challenge · Official Cycle 1

Ahead of Mem0 and Supermemory on the coding track.

The organizers list Claude-Mem v12.1.1 second among 15 commercial-product entries. Its 52.00% overall score put it ahead of mem0-api v3 at 51.33% and SuperMemory v3write/v4search at 49.33%.

Test conditions and comparison

Claude-Mem’s submitted API resolved 56.86% of new-feature tasks and 49.49% of bug-fix tasks. The organizers rated both input and output token usage low. MemoraX led at 62.00%. Two other entries shared Claude-Mem’s 52.00% overall score. The resource ratings are categories, not token counts.

55 / 64Fresh handoffs fully passed
Crosshead · Competing vendor’s test

Carry decisions into a different coding agent.

Crosshead tested local session continuity with Claude Code and Codex. Claude-Mem v15.12.1 passed 55 of 64 fresh handoffs in full. Every Claude Code-to-Codex handoff passed in both suites, including 16 of 16 fresh runs and 12 of 12 main-suite runs.

Test conditions and comparison

Crosshead used seeded tasks, four transfer directions and a Haiku observer for Claude-Mem. The 95% interval for the fresh full-pass rate was 75%–92%. Crosshead’s own system scored higher. The authors published their grader criteria, costs and individual runs.

3 / 3Fresh sessions recovered the rule
Ishank Choudhary · Independent hands-on test

Recover the business rule that only existed in chat.

In OpenCode, three fresh sessions using Claude-Mem v15.18.1 recovered a chat-only discount policy. Three clean sessions without memory recovered it zero times. Restating the rule manually also worked three times.

Test conditions and comparison

This was one task repeated three times per condition. The first session saved the rule in a note. Each memory run began with the same database snapshot and retrieved the note by search. The publisher measured 126,307 context tokens per memory run, 88,428 without memory and 65,739 with the rule re-explained. These totals include cached input.

MatchedSessions with and without Claude-Mem
WPP Research · Hands-on evaluation

Remember why a feature was removed.

WPP Research tested Claude-Mem in Claude Code. With memory enabled, the agent recovered earlier session summaries and the rationale for removing a feature. Without it, the agent inferred an explanation from the remaining code.

Test conditions and comparison

The code already showed the retry pattern. Encouraging memory search on that task took 46 seconds and 2,400 output tokens. The run without that instruction took 27 seconds and 1,300 output tokens. WPP also found that two folders with the same name shared context in its tested setup.

Production experience

What does a working deployment look like?

One developer published 70 days of Claude-Mem capture across three machines, including the cost of observation and what it took to keep search working.

Read the 70-day case study

Claude-Mem

Give your next session a head start.

We capture the decisions and fixes while you build. CMEM Pro puts that memory within reach of your connected tools.