Diagram

How every system was tested

  1. 1. Same data85 memoriesNotes about one user and one project, written in 6 sessions. 15 facts change in later sessions.
  2. 2. Same callsOne adapter eachA short script calls the project's own write and search API with its default settings.
  3. 3. Same questions75 recalls, top 525 with the same words, 25 reworded, 15 about changed facts, 10 near misses.
  4. 4. Same scoringOne score scriptDoes a returned memory hold the answer? For a changed fact, does the new value rank above the old?
Every adapter, the data set and the scoring script are in benchmarks/agent-memory in the XERJ repository. Systems that need embeddings all used all-MiniLM-L6-v2 on CPU, so the model is the same and the design is what differs.