membench: what a memory layer is worth, and where it is not worth anything

First post here, so it should be the house rule in action rather than a description of it: a benchmark of my own memory layer, published with every column where it loses, one section listing the numbers I withdrew, and one listing what the corpus cannot support.

Code, data and the five-page report: https://github.com/polmanas1998-star/membench

Design

Recall is the easy half. Four things measured together, on one corpus, with the base model held constant so the memory is the only variable: retention of a fact stated once and long ago, supersession where a fact was updated and the stale value must not come back, age where an old answer must carry a caveat, and absence where nothing was ever stated and the correct output is a refusal.

Three layers of control

Borrowed openly. HELM refuses a single headline score. Tau-bench scores over repeated trials rather than one run. GPQA and ARC-AGI validate that the task is hard before reporting who solved it.

1. Witnesses validate the harness: one arm that never answers, one that always guesses, one that is perfect. If those three do not return three distinguishable verdicts, nothing is published. A precision computed on zero answers is n/m, never 1.000, or silence wins the benchmark.
2. A trivial baseline validates that there is something to beat: always answer the most frequent object.
3. A shortcut control validates the task: the same layer, fed a timeline whose objects were randomly redistributed. If it still scores, the questions were answerable without memory.

Results, 832 questions over 8 seeds

Medians, d = 2048. Lower is better on stale-err and halluc.

arm                 cover  gatedP  retain  stale-err   age  deep-ret  halluc
oracle              0.788   1.000   1.000      0.000  1.000    1.000   0.000
majority baseline   1.000   0.062   0.071      0.056  0.000    0.071   1.000
memory layer        0.500   1.000   1.000      0.000  1.000    0.000   0.000
layer [scrambled]   0.524   0.043   0.036      0.000  1.000    0.000   0.000

Paired bootstrap over the 832 questions, 4000 draws, 95% intervals, on gated precision:

layer minus scrambled twin      +0.959  [+0.939, +0.977]
layer minus trivial baseline    +0.933  [+0.915, +0.950]
layer minus perfect oracle      -0.005  [-0.012, +0.000]

The last band touches zero: the shortfall against a perfect oracle is not distinguishable from sampling noise at this size. That is not the same as being perfect.

The uncomfortable result

The scrambled control keeps coverage at 0.524, slightly above the honest layer's 0.500. It answers just as often while being wrong 96% of the time, and the confidence gate does not notice.

The gate measures internal consistency, not correctness. Fed well-formed false memories, the layer is exactly as confident. No internal signal fixes that; only corroboration against something outside the trace can.

Where the errors are

Outcome by question class, 8 seeds:

class         correct  silence  wrong  stale-value  hallucinated      n
stable            144        0      0            0             0    144
reinforced         80        0      0            0             0     80
superseded        112       32      0            0             0    144
faded              75      101      0            0             0    176
expired             0      112      0            0             0    112
absent              0      174      0            0             2    176
TOTAL             411      419      0            0             2    832

On the 368 questions the system should be able to answer: 336 correct, 0 wrong. Silence rises monotonically with how little it is entitled to know.

This table paid for itself. The first pass showed 3 wrong answers on facts past the forgetting threshold, a class where the layer had zero correct out of 112. Past that line the trace no longer holds the fact; what remains is noise, and the gate scores noise the way it scores a memory. Refusing the class outright cost no correct answer and removed three of five errors.

A confidence gate and an age gate are not the same thing. One asks how sharp a memory is, the other whether it has any right to exist. A system with only the first is confident about its own decay products.

The gate is a dial, not a number

Saying the layer answers half the questions describes one setting. 5 seeds:

z     coverage  gated-precision  hallucination
1.0      0.865            0.711          1.000
2.0      0.798            0.800          0.682
3.0      0.596            0.952          0.091
4.0      0.490            1.000          0.000
5.0      0.423            1.000          0.000
8.0      0.250            1.000          0.000

The knee is sharp. Between z = 2 and z = 4 the layer trades a third of its coverage for two tenths of precision, and hallucination on facts never stated falls from 0.682 to zero.

What is lost by design

Deep retention is 0.000 and does not move at d = 1024, 2048 or 4096. A fact leaves the trace at 111.33 days, which is 45 x log2(1/0.18) from the half-life and the weight floor; measured effective weight is 0.1809 on day 111 and 0.1781 on day 112. This is decay, not capacity. On any fact older than that, a system that simply keeps the transcript recovers everything and this one recovers nothing.

Withdrawn, and not yet measured

Withdrawn. A cost comparison against a full-transcript arm and a top-k retrieval arm was measured on 24 questions and is not published here. Those runs used a candidate list sorted identically in every question, so a model's position preference could not be separated from its memory. The confound was found and fixed after the measurement; the numbers will be re-taken, not amended.

Not yet measured. Those same arms across seeds at full corpus size. The free tier allows 200,000 tokens a day, and the measured refill after exhausting it is 663 tokens an hour, about one turn an hour, so a campaign has to start a day after the last one ends.

External validity

The corpus is synthetic: 104 generated subject-relation-object facts about a fictional architecture practice, on a simulated timeline. That buys exact ground truth, a known denominator for every class, and byte-for-byte reproducibility per seed. It does not buy natural language, realistic fact distributions, a second domain, a second base model, or real ageing. Full section in the report.

What it does support are statements about the layer's own arithmetic under known conditions. Everything comparative is a hypothesis to be re-tested on real transcripts.

Harness

40 tests, 13 mutants of 13 killed. Two tests were decorative until mutation found them, including one that guarded a counter without ever exercising the code that sets it. The significance test is itself guarded by comparing an arm against itself and requiring the interval to contain zero: the first implementation failed that, reporting a difference of exactly zero as significant.

Happy to be told the numbers are wrong.

---

Figures and full report

• Figure 1, the three controls
• Figure 2, the precision and coverage curve
• Figure 3, outcome by question class
• The five-page report, PDF

Reproduce the offline table yourself, no API key, about a minute:

pip install numpy
export HOLOMEM_PATH=/path/to/holomem
python sweep.py --seeds 8