Nex Local Bench
Reproducible LongMemEval harness for local LLMs against memory systems
Run LongMemEval against any memory system (Nex, Mem0, Graphiti, Letta) with a local LLM reader (Qwen, Llama via Ollama). 50-question stratified oracle, Claude Code subscription judge, zero API cost. Python, MIT.
What it solves
The memory-backed model space has a credibility problem. One system published 49 percent on LongMemEval in 2025, then 93.4 percent in April 2026 on a variant nobody else can reproduce. Another claimed 96.6 percent in a paper that was never peer reviewed. A third put 95.4 percent in its marketing and 76.8 percent in its own bench report. Peer-reviewed, reproducible numbers cluster somewhere around 65 to 85 percent. On top of that, every published score uses a cloud reader, including from systems that call themselves local-first.
How it works
An open evaluation harness with the scripts, the configs and the seeds in the repository, so a number can be re-run rather than believed. It drives local readers, Qwen 3 and Llama 3.1, against four memory systems: a self-hosted Mem0, Graphiti, Letta and StudioMeyer Memory. The 50-question oracle set runs as a pre-flight and the 500-question set as the full run.
When to use it
Use it when you need to compare memory systems for yourself instead of comparing their marketing. The value is not our number, it is that you can produce your own on your own hardware with your own reader and see the spread.
When not to use it
A benchmark is a measurement under one set of conditions, not a ranking that settles anything. LongMemEval tests long-conversation recall and says nothing about latency, cost, operational effort or how a system behaves on your actual data. If you take a single number from here and put it on a slide, you are doing the thing this repository exists to argue against.
Stars
★ 0
Clones (14d)
⊟ 3
Language
Python
Updated
2026-08-03