---
title: "Nex Local Bench · Reproducible LongMemEval harness for"
description: "Reproducible LongMemEval harness for local LLMs against memory systems. Run LongMemEval against any memory system (Nex, Mem0, Graphiti, Letta) with a local LLM"
canonical: "https://matthiasmeyer.tech/repos/nex-local-bench"
language: "en"
last_updated: "2026-08-03"
markdown_versions: ["https://matthiasmeyer.tech/repos/nex-local-bench.md", "https://matthiasmeyer.tech/de/repos/nex-local-bench.md", "https://matthiasmeyer.tech/es/repos/nex-local-bench.md"]
image: "https://matthiasmeyer.tech/api/og/repo/nex-local-bench?lang=en"
publisher: "Matthias Meyer, https://matthiasmeyer.tech (llms.txt: https://matthiasmeyer.tech/llms.txt)"
---

memory alpha Python

# Nex Local Bench

Reproducible LongMemEval harness for local LLMs against memory systems

Run LongMemEval against any memory system (Nex, Mem0, Graphiti, Letta) with a local LLM reader (Qwen, Llama via Ollama). 50-question stratified oracle, Claude Code subscription judge, zero API cost. Python, MIT.

[View on GitHub →](https://github.com/studiomeyer-io/nex-local-bench)

## What it solves

The memory-backed model space has a credibility problem. One system published 49 percent on LongMemEval in 2025, then 93.4 percent in April 2026 on a variant nobody else can reproduce. Another claimed 96.6 percent in a paper that was never peer reviewed. A third put 95.4 percent in its marketing and 76.8 percent in its own bench report. Peer-reviewed, reproducible numbers cluster somewhere around 65 to 85 percent. On top of that, every published score uses a cloud reader, including from systems that call themselves local-first.

## How it works

An open evaluation harness with the scripts, the configs and the seeds in the repository, so a number can be re-run rather than believed. It drives local readers, Qwen 3 and Llama 3.1, against four memory systems: a self-hosted Mem0, Graphiti, Letta and StudioMeyer Memory. The 50-question oracle set runs as a pre-flight and the 500-question set as the full run.

## When to use it

Use it when you need to compare memory systems for yourself instead of comparing their marketing. The value is not our number, it is that you can produce your own on your own hardware with your own reader and see the spread.

## When not to use it

A benchmark is a measurement under one set of conditions, not a ranking that settles anything. LongMemEval tests long-conversation recall and says nothing about latency, cost, operational effort or how a system behaves on your actual data. If you take a single number from here and put it on a slide, you are doing the thing this repository exists to argue against.

Stars

★ 0

Clones (14d)

⊟ 3

Language

Python

Updated

2026-08-03

## Sitemap

Every page of this site as Markdown: [sitemap](https://matthiasmeyer.tech/sitemap.md). Curated entry point: [llms.txt](https://matthiasmeyer.tech/llms.txt).