The strongest result on difficult issues with repository history.
Chen and colleagues ran five memory systems through the same coding harness. With a history of earlier repository work, Claude-Mem led the five on 133 selected difficult SWE-bench Verified tasks. The no-memory agent resolved 35.3%. Claude-Mem also led with relevant-only history at 41.6%.
Test conditions and comparison
The researchers pinned Claude-Mem v12.1.1 and used Claude Code with DeepSeek-V4-Flash. They froze each history and ran each target three times. On the difficult subset, Claude-Mem resolved 40.1% with noisy history, where Supermemory reached 40.6%. On 50 selected easier tasks, repository history gave Claude-Mem 96.7% versus 94.0% without memory. The subsets were selected using separate baseline runs.