Applies to OML v0.6.x · Status: Measured · Last verified 2026-11-06
These are measurements by the OML maintainers on their own hardware, not independent results. Generated from the committed oml-bench/1 results.
Suite beir-nfcorpus, data beir/nfcorpus/test (ir_datasets 0.6.3): 3,633 memories, 323 queries; a method marked sampled asked a seeded subset, named in its row.
How to reproduce. Build oml and oml-server-lead in release mode, install eval/bench/requirements.txt in a scratch virtual environment, then run eval/bench/run.sh <out-dir> nfcorpus against a throwaway PostgreSQL superuser cluster. The harness loads the data over HTTP, promotes it with reviewer keys, asks every query through oml_read and writes the result files listed below.
Reference: BEIR BM25 (Anserini, k1 0.9, b 0.4, title and text fields), ndcg@10 0.325 (source).
Chart: nDCG@10 by pipeline
Latency
Client wall time per recall, in milliseconds, nearest-rank percentiles.
method
p50
p95
p99
keyword-all
477
513
537
keyword-any
586
1,016
1,072
bm25
51.8
83.8
106
bm25-trigram
2,172
3,047
3,391
bm25-postings
38.0
54.8
77.2
sparse
35.7
49.4
63.6
bm25-sparse
48.5
65.9
78.8
bm25-sparse-rrf
34.4
58.7
182
bm25-rerank-d20
5,247
15,467
16,744
bm25-sampled
69.5
140
197
reference-bm25s-k1-0.9-b-0.4
offline
offline
offline
reference-bm25s-k1-1.2-b-0.75
offline
offline
offline
Chart: p95 latency by pipeline
What this result does not show
NFCorpus is nutrition and medical text. It does not predict recall over short chat notes.
Limits of this run
One abstract per memory as '
. <text>' (BEIR indexes two fields); k 100, no token budget; English only. Known gap: served bm25 measures below BEIR's published 0.325 and Pyserini's single-field 0.322; the offline textbook BM25 over the same memories measures 0.318, so most of the gap is the one-body mapping and the analyser (Snowball against Lucene's Porter, hyphenated words kept whole), not the ranking.</p>
<ul>
<li>Offline bm25s over the same memories (Lucene BM25, English stop words, Snowball stems), not a served path;</li>
<li>Served bm25 over the same seeded sample, on the same host in the same hour, for comparison with bm25-rerank-d20; the full-set bm25 row is the reference run.</li>
<li>Known gap: served bm25 measures below BEIR's published 0.325 and Pyserini's single-field 0.322; the offline textbook BM25 over the same memories measures 0.318, so most of the gap is the one-body mapping and the analyser (Snowball against Lucene's Porter, hyphenated words kept whole), not the ranking. bm25's list reranked by ms-marco-MiniLM-L6-v2 through candle on the CPU at depth 20 (the default became 10 after this run), with a cap of 8 and a 60 s deadline so no rerank was skipped; the host carried a load average of 9 to 29 from other work, and the LoCoMo run shared it with the NFCorpus run, so its latency is an upper bound (.</li>
<li>Known gap: served bm25 measures below BEIR's published 0.325 and Pyserini's single-field 0.322; the offline textbook BM25 over the same memories measures 0.318, so most of the gap is the one-body mapping and the analyser (Snowball against Lucene's Porter, hyphenated words kept whole), not the ranking. bm25-postings takes its candidates from the postings of the query forms it reads; when a query's readable postings pass POSTING_BUDGET (10,000) it leaves out the forms held by more than 10% of the readable rows, so a row holding only those is no candidate, and below the budget it reads every form and ranks as bm25.</li>
</ul>
<p>Sources: <code>eval/bench/results/beir-nfcorpus/keyword-all.json</code>, <code>eval/bench/results/beir-nfcorpus/keyword-any.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-trigram.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-postings.json</code>, <code>eval/bench/results/beir-nfcorpus/sparse.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-sparse.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-sparse-rrf.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-rerank-d20.json</code>, <code>eval/bench/results/beir-nfcorpus/bm25-sampled.json</code>, <code>eval/bench/results/beir-nfcorpus/reference-bm25s-k1-0.9-b-0.4.json</code>, <code>eval/bench/results/beir-nfcorpus/reference-bm25s-k1-1.2-b-0.75.json</code>.</p>
</article>
</div>
<script>var tabs=__md_get("__tabs");if(Array.isArray(tabs))e:for(var set of document.querySelectorAll(".tabbed-set")){var labels=set.querySelector(".tabbed-labels");for(var tab of tabs)for(var label of labels.getElementsByTagName("label"))if(label.innerText.trim()===tab){var input=document.getElementById(label.htmlFor);input.checked=!0;continue e}}</script>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
</div>
<button type="button" class="md-top md-icon" data-md-component="top" hidden>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg>
Back to top
</button>
</main>
<footer class="md-footer">
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../beir-scifact/" class="md-footer__link md-footer__link--prev" aria-label="Previous: BEIR SciFact">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg>
</div>
<div class="md-footer__title">
<span class="md-footer__direction">
Previous
</span>
<div class="md-ellipsis">
BEIR SciFact
</div>
</div>
</a>
<a href="../locomo/" class="md-footer__link md-footer__link--next" aria-label="Next: LoCoMo">
<div class="md-footer__title">
<span class="md-footer__direction">
Next
</span>
<div class="md-ellipsis">
LoCoMo
</div>
</div>
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11z"/></svg>
</div>
</a>
</nav>
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">
<div class="md-copyright__highlight">
Code licensed <a href="https://github.com/e8kor/open-memory-lab/blob/main/LICENSE">Apache-2.0</a> · documentation licensed <a href="https://github.com/e8kor/open-memory-lab/blob/main/LICENSE-docs">CC BY 4.0</a>
</div>
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<div class="md-dialog" data-md-component="dialog">
<div class="md-dialog__inner md-typeset"></div>
</div>
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": ["navigation.tabs", "navigation.tabs.sticky", "navigation.indexes", "navigation.path", "navigation.footer", "navigation.top", "navigation.instant", "search.suggest", "search.highlight", "toc.follow", "content.tabs.link", "content.code.copy", "content.action.edit", "content.action.view"], "search": "../../assets/javascripts/workers/search.a7633da7.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.c21346df.min.js"></script>
</body>
</html>