Benchmarks

We measure before we claim. And we publish our failures.

Most memory systems can store facts. The MEME benchmark (KAIST AI · Tübingen · NAVER, 2026) — the first evaluation built around evolving organizational memory — showed that storage was never the hard part. Across 100 episodes and six tasks, practical-cost systems handle exact recall and aggregation reasonably. Then the benchmark asks two questions that organizations ask every day, and the field collapses.

The two tasks that matter

Cascade — when a fact changes, does everything that depends on it change too?
A vendor's contract is renegotiated. The budget line that referenced the old terms, the project plan built on the old timeline, the commitment made downstream — do they update, or does the system keep answering from a world that no longer exists? Field average at practical cost: 3%.

Absence — does the system know what it doesn't know?
The most dangerous output of a memory system is not a wrong answer; it is a confident restatement of a stale fact. Absence measures whether a system can say "this is no longer supported" instead of fluently asserting yesterday. Field average at practical cost: 1%.

These numbers are the benchmark authors', not ours, and they are the honest state of the category. Behind the marketing, almost every memory system in production today fails the two tasks that define organizational reality: things change, and knowing that you don't know is worth more than sounding sure.

Why N71 should be tested exactly here

These two tasks are not incidental to our architecture — they are what it was built for. Cascade is a supersession problem: N71 records facts with temporal lifecycle, invalidates rather than deletes, and links dependent assertions through an explicit graph (TR-2026-01 §3.4). Absence is a refusal problem: N71's answer protocol validates every citation against anchored sources, strips what it cannot verify, and returns an explicit, machine-readable refusal when the corpus cannot support an answer — by design, it fails closed (TR-2026-01 §5).

A system that claims those properties should be willing to be measured on them. We are.

Our results

Runs of June 12–16, 2026 — the full 100-episode suite, the benchmark's published episodes and judges, through our production pipeline. Three protocol iterations, all shown.

All three N71 rows are the full 100-episode suite, scored by the benchmark's published judge on the strict metric the paper uses for the dependency tasks — credit only when the system knew the fact before the change and handled it correctly after (MEME §4, trivial-pass filtering). Iteration 3 is now our standing result and the figures we cite; iterations 1 and 2 are kept in full, including iteration 2's regression, so the trajectory is legible. Bold marks the standing best per task — now iteration 3 on Tracking, Cascade, Absence, and Overall (Deletion's best remains iteration 1's 0.56). Baseline rows: MEME, Table 2 (Jung et al., arXiv:2605.12477), all run end to end on gpt-4.1-mini. The 70× reference is the study's frontier-model file agent, which its authors describe as not deployable today. Run artifacts for all three iterations are published below for reproduction.

What the table says. Iteration 1 is the highest overall accuracy of any memory system in the study — above the best published system (0.42), above both no-memory full-transcript baselines, and within 0.04 of the 70× frontier reference (0.55) at roughly 1/70th the cost. The margins sit exactly where the field collapses: Cascade at 0.55 against a 3% field average and a 6% prior best — nine times the prior state of the art — Absence at 0.35 against a 1% field average, and Deletion at 0.56, more than double the previous best. Every dependency-task pass is a strict real pass: the system knew the fact before the change and handled it correctly after.

What iteration 2 says — published in full, including a regression. We said in our last run that we would fix Aggregation, and we did: 0.12 → 0.29, more than double, by capturing membership and affiliation facts onto the entity's card instead of leaving them stranded as graph edges the answer path never read. But the same release traded ground on the tasks that define this benchmark: Cascade fell 0.55 → 0.48 and Deletion 0.56 → 0.50. Overall edged up to 0.522 — a 0.010 move that is within run-to-run noise — so we do not count iteration 2 as progress. It moved a task we cared less about and went backwards on the two we care about most. We have diagnosed the cause (a field-naming drift introduced by the same change — the new value lands on a sibling field, so a cascade rule fires against a field the answer no longer reads) and the fix is in progress. Iteration 1 remains the result we cite until a run beats it on Cascade and Absence, not just on the average.

What iteration 3 says — the regression is recovered, and through a mechanism that ships. We named the iteration-2 cause as field-naming drift, and iteration 3 fixes it at the root. Cascade recovers to 0.628 — past iteration 1's 0.55, against a 3% field average — and Absence to 0.42, both new bests, while Aggregation holds its iteration-2 gain. The fix is read-time dependency resolution over canonical field slots: a canonical-slot layer that recognizes when different surface names mean the same fact (conservatively — with guards that never merge genuinely distinct fields, the exact failure that caused the iteration-2 regression), and a read-time resolver that reconnects a dependency to the parent that actually changed, using the workspace's own dependency structure rather than any benchmark-specific knowledge. Deletion is held with a tombstone-aware canonicalization that stops a drifted re-mention from resurrecting a deleted fact. Overall 0.574 — above iterations 1 and 2, and the highest in the study. A note on reading the Cascade figure: like every number here it is a single full-suite run, and on a parallel run with the Deletion fix bundled it landed at 0.585 — so read Cascade as "recovered well past 0.55," robust to run-to-run variance, rather than to the third decimal.

Where we lose, published on purpose. Three losses, each with its mechanism named.

Aggregation (0.12 in iteration 1) was our worst number, and we knew exactly why. Every Aggregation question asks for a complete multi-part list — hobby, sport, and club membership. Iteration 1 reliably surfaced the first two and dropped the third: membership facts were captured as relationships in the graph (User → member_of → Book Club) rather than as facts on the user's card, so the answer surface assembled two of three and never saw the rest. Retrieval wasn't failing — the fact was filed in a drawer the answer path didn't read for this question shape. The fix was specific and already designed: fold relationship-shaped personal facts (memberships, affiliations) into the card's attribute capture as accumulating set entries, and surface member_of edges into answer context. Iteration 2 shipped that fix and it worked — Aggregation 0.12 → 0.29. We are publishing it alongside iteration 1, not replacing it, because the same release regressed Cascade and Deletion; that trade, and the field-drift cause behind it, is described above and is the focus of the next iteration.

Tracking (0.50) trails the study's best file agent (0.77) — full version-chain retrieval is an iteration target.

Absence (0.35) is the best practical-cost result in the study, and it is not where we want it. A note on how it got here, because the path is a lesson in sample sizes: in six-episode iteration runs, Absence moved from 0.22 to 0.89 as we shipped conditional refusal — derive when a dependency rule determines the new value, refuse when none does. On the full 100-episode suite it lands at 0.35. That spread is why we run the full suite before claiming anything — and why any memory benchmark number published without a disclosed sample size and methodology should be read accordingly. The full analysis of the propagation–refusal trade is in TR-2026-01 §5.

The ceiling — and where optimizing stops being worth it

We said we would publish the benchmark's in-context upper bound. Here it is. Give the answer model the gold facts directly, no retrieval, and ask the same questions: it measures how high any memory system could score with this answerer — the task-solvability ceiling. Run on all 100 episodes, same methodology (gpt-4.1-mini answerer, gpt-4o judge).

Two things this settles. With good memory the answerer nails Cascade, Tracking, and Exact Recall (1.00) — those are pure memory problems, and the remaining gap there is ours to close. But Aggregation caps at 0.54 even with perfect facts (0.22 on the software half): the fixed answer model cannot reliably enumerate a complete "tell me everything" list no matter how good the memory is. That ceiling is a property of the benchmark's mandated answerer — a model most of our customers do not run — not of the memory. It is the clearest place where further optimization stops being worth it.

Where we're headed next

We are pausing benchmark optimization here, deliberately. The ceiling shows the core memory capability is sound — with good memory the answerer scores in the high-0.8s, and our dependency-task results (Cascade, Absence, Deletion) now lead the field at practical cost. The headroom that remains is two things we judge not worth chasing on this benchmark: distractor-robustness under 32k tokens of injected noise (a stress condition real workspaces rarely hit), and the fixed gpt-4.1-mini answerer's own ceiling on Aggregation (a model our customers do not use). Our engineering returns to the product; we will publish again if a change moves the dependency tasks materially.

How we run it — and what we'll publish

The benchmark's own rules. The full 100-episode suite, scored with the benchmark's published judge prompts, sessions ingested in order, no task-specific tuning. The harness is an adapter over the same ingestion and answer paths our customers use — we benchmark the product, not a lab build.

Every failure, classified. For each miss we publish the cause: retrieval miss (the evidence was never surfaced), ranking loss (surfaced but out-ranked), or reasoning failure (surfaced, ranked, and wrongly synthesized). A score tells you where a system is; the failure taxonomy tells you whether its architecture can get better. Iteration 1's Cascade misses: 64 reasoning failures, 4 ranking losses, 2 retrieval misses. Iteration 2's: 77 reasoning failures, 5 ranking losses, 3 retrieval misses. In both runs retrieval is effectively solved — the evidence is almost always surfaced — and the remaining work is in the protocol that decides what to do with it. That is precisely why the iteration-2 regression is recoverable: nothing stopped being found; a rule started firing against the wrong field name.

The numbers we're proudest of and the ones we're not. A vendor that publishes only its wins should be presumed to have losses. Our results page will carry both, permanently, with the run methodology alongside.

Run artifacts

Every number above is recomputable from the raw results. No selection, no summarization — all 1,188 questions per iteration, passes and failures alike.

Reproduce it yourself — [github.com/syags13/n71-meme-benchmark](https://github.com/syags13/n71-meme-benchmark). Run python score.py results/ to recompute the 0.574 headline from all 1,188 judged questions in about ten seconds: no install, no keys, no network — the scorer is the aggregation, in the open. Re-judge our answers with your own gpt-4o key, or request a key to re-run N71 end to end. The dataset and judge are the MEME authors' published versions (arXiv:2605.12477).

Iteration 1 · June 12, 2026
Complete results — Excel — every question: episode, task, the question, the gold answer, N71's answer, the judge's verdict, and the real/trivial classification. Three sheets: summary, all questions, per-episode.
Complete results — CSV · JSON — the same data for programmatic use.
Per-episode breakdown — CSV — per-task pass counts for each of the 100 episodes.

Iteration 2 · June 13, 2026 (membership capture + dependency recall)
Complete results — Excel — same schema; this is the run with Aggregation 0.29 and the Cascade/Deletion regression, every question included.
Complete results — CSV · JSON — the same data for programmatic use.
Per-episode breakdown — CSV — per-task pass counts for each of the 100 episodes.

Iteration 3 · June 16, 2026 (read-time dependency resolution + canonical slots)
Complete results — Excel — same schema; the run with Cascade 0.628 / Absence 0.42 / Overall 0.574, every question included.
Complete results — CSV · JSON — the same data for programmatic use.
Per-episode breakdown — CSV — per-task pass counts for each of the 100 episodes.

Episodes and judge prompts are the benchmark's published versions (Jung et al., arXiv:2605.12477). Answer model: gpt-4.1-mini, the study's standard. Judge: gpt-4o. Internal identifiers and infrastructure diagnostics are scrubbed from the export; the benchmark substance — every question, answer, and verdict — is complete.

A note on reading memory benchmarks

Two things to check before believing any number in this category — including ours. First, who scored it: self-reported results on a benchmark the vendor selected deserve the same scrutiny as any other marketing claim; methodology should be published in enough detail to rerun. Second, the frontier-model baseline: on some tasks, a raw frontier model with the transcript in context already scores respectably — a memory system has to beat the baseline of simply not having a memory system, at a cost that makes sense. We hold ourselves to both checks.

---

Reproduce or validate every number: [github.com/syags13/n71-meme-benchmark](https://github.com/syags13/n71-meme-benchmark). Methodology questions, or a key to re-run N71 end to end: sanad@n71.ai