12 setups, 75 questions each
| System | Setup | Top 1 % | Top 5 % | Reworded % | Newest first % | Write ms | Query ms | LLM calls per write | Packages |
|---|---|---|---|---|---|---|---|---|---|
| XERJ | BM25, default recall | 54.7 | 73.3 | 24.0 | 13.3 | 11.3 | 3.8 | 0 | 0 |
| XERJ | semantic, MiniLM | 65.3 | 94.7 | 84.0 | 13.3 | 7.2 | 6.3 | 0 | 0 |
| XERJ | hybrid, MiniLM | 60.0 | 93.3 | 84.0 | 13.3 | 6.5 | 6.5 | 0 | 0 |
| XERJ | hybrid + recency 0.3 | 44.0 | 68.0 | 36.0 | 93.3 | 6.5 | 6.8 | 0 | 0 |
| Mem0 | infer=False, MiniLM | 64.0 | 88.0 | 68.0 | 13.3 | 10.2 | 22.4 | 0.0 | 50 |
| Mem0 | infer=True, Qwen2.5 3B | 64.0 | 86.7 | 72.0 | 20.0 | 16523 | 29.1 | 1.0 | 50 |
| MemPalace | default, MiniLM | 57.3 | 84.0 | 56.0 | 13.3 | 21.2 | 22.7 | 0 | 86 |
| ReMe | default, BM25 | 53.3 | 70.7 | 20.0 | 20.0 | 2.7 | 1.4 | 0 | 138 |
| ReMe | hybrid, MiniLM | 61.3 | 93.3 | 80.0 | 13.3 | 2.6 | 12.0 | 0 | 138 |
| agentmemory | default, no key | 53.3 | 74.7 | 28.0 | 13.3 | 4.4 | 4.8 | 0 | 143 |
| mcp-memory-service | default, MiniLM | 64.0 | 89.3 | 84.0 | 6.7 | 16.7 | 16.7 | 0 | 72 |
| memvid | default, lexical | 1.3 | 1.3 | 0.0 | 0.0 | 67.3 | 0.3 | 0 | 2 |