90.8%
454 of 500 questions answered correctly
90.0%
on the 440 questions never used to tune chatcli
469 / 470
questions where chatcli’s search surfaced the evidence
22 s · $0.08
median time and average cost per question
The story behind the numbers
The article on LinkedIn walks through all 500 questions, the real failures, the bugs the run found and what changed afterwards. Written in Portuguese.
What LongMemEval is
LongMemEval is a benchmark from the paper LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang and Dong Yu, ICLR 2025). It has 500 hand-curated questions. Each question comes with its own timestamped history of chat sessions. In the S variant that history is about 115k tokens, roughly 40 to 50 sessions of everyday conversation, and the evidence for the answer sits somewhere inside. The questions test five abilities:Why it matters
- It is hard on purpose. The paper reports that commercial chat assistants and long-context LLMs show a 30% accuracy drop on memorizing information across sustained interactions. A bigger context window alone does not solve it.
- It tests the failure users actually see. An assistant that forgets a correction, mixes up two trips or invents an answer it was never given loses trust fast. LongMemEval scores each of those separately.
- It is reproducible. The dataset, the questions, the gold answers and the judge prompts are public (MIT), so anyone can rerun it and compare.
How chatcli was tested
The goal was to test chatcli the way you use it, not to test the model. Nothing was pasted into the prompt. Each question’s history became chatcli’s own saved sessions, and chatcli had to find the evidence through its memory tools.1
One isolated installation per question
A fresh
CHATCLI_STATE_DIR holds the question’s history as saved sessions; each file’s modification time is the session’s date. Your real memory never mixes in, and your login is reused in place through CHATCLI_AUTH_DIR.2
The clock set to the question's day
CHATCLI_NOW is the question date, so “two weeks ago” means two weeks before the question, not two weeks before the run.3
chatcli answers by itself
chatcli -p "/coder …" with Claude Sonnet 5.5 and one neutral instruction: answer from our previous conversations, say so if they do not contain the answer. chatcli decides which tools to call.4
The official judge
Each answer is graded with the official LongMemEval answer-check prompts, unchanged, run with Claude Sonnet 5.5.
Results
454 of 500 correct: 90.8% (95% confidence interval 87.9%–93.0%). Averaged across the six question types, so each type weighs the same: 92.2%.Held-out score
Before the full run, a 60-question sample (10 per type, fixed seed) was used to find and fix weaknesses in chatcli. Those 60 scored 96.7% in the final run, so they are not a neutral test. On the other 440 questions chatcli scored 396/440 = 90.0% (95% CI 86.8%–92.5%), with the same 92.2% task average. That is the conservative number.Where it fails
The run records, for every question, which saved sessions hold the answer according to the dataset’s labels and whether those sessions reached the model. That separates finding from reasoning.
- Counting across sessions (multi-session, 22 misses): “How many projects have I completed since I started painting classes?” — chatcli found 4 of the 5.
- Date arithmetic and ordering (temporal, 16 misses): “Which book did I finish first?” when one session says “last weekend” and another “just finished”.
- Too cautious: in 9 questions the evidence was in front of the model and it answered “that is not in our conversations”.
- Too eager: in 3 of the 30 unanswerable questions it guessed instead of abstaining.
Two of the 46 misses had a different cause, found by reading the transcripts: a search for “St. Mary’s Church” carried an unescaped apostrophe that cut the tool call short. That parser bug is fixed in #1842, and both questions pass when rerun. The 90.8% above is the score as measured, without that fix.
Cost and speed
chatcli used
@session search 1,068 times, @session get 967 times and @memory recall 344 times. In 66 questions it narrowed searches by date on its own, with the from, to and as_of flags of session search (137 calls). It never went to the web.
What the benchmark changed in chatcli
Running LongMemEval found real bugs, and each one was fixed in chatcli itself, not in the test:
On the 60-question sample, these changes took temporal reasoning from 8/10 to 10/10, knowledge updates from 8/10 to 9/10 and multi-session from 9/10 to 10/10, with no question going the other way.
Read the numbers honestly
Not directly comparable with GPT-4o-judged scores
Not directly comparable with GPT-4o-judged scores
The paper’s reference judge is GPT-4o. This run used the same official prompts with Claude Sonnet 5.5 as the judge, so the score is not directly comparable with leaderboard numbers judged by GPT-4o.
The 60 tuning questions
The 60 tuning questions
The 60-question sample guided the fixes above. The held-out 90.0% on the other 440 is the number that was never seen while tuning.
One run per question
One run per question
Models vary between runs. A question near the edge can flip either way on a rerun.
What 'evidence surfaced' means
What 'evidence surfaced' means
An evidence session’s name appeared in what the model saw: search results, the auto-recall block or a page it read. It shows retrieval reached the right conversation, not that the model read the exact sentence.
Session names carry dates
Session names carry dates
Saved sessions were named
chat-<date>-<index>; the dataset’s own ids would have revealed which sessions hold the answer. The date in the name is the same information every search hit shows.The data
Everything from the run is public in the chatcli repository, underevals/longmemeval: