Skip to content

BEIR SciFact

Applies to OML v0.6.x · Status: Measured · Last verified 2026-11-06

These are measurements by the OML maintainers on their own hardware, not independent results. Generated from the committed oml-bench/1 results.

Suite beir-scifact, data beir/scifact/test (ir_datasets 0.6.3): 5,183 memories, 300 queries; a method marked sampled asked a seeded subset, named in its row.

Run details

Measured 2026-11-05 on commits 2123e90, 66979b4, a1ad0c9, d3fdb72, f86c31c; build release, host 8 vCPU / 15 GB, postgres 17.9; build release, host 8 vCPU / 15 GB, postgres 18.6; build release, host 8 vCPU / 15 GB, load average 3.6 to 14 from other lanes, postgres 18.6; build release, host 8 vCPU / 15 GB, shared, load 8-12, postgres 18.

How to reproduce. Build oml and oml-server-lead in release mode, install eval/bench/requirements.txt in a scratch virtual environment, then run eval/bench/run.sh <out-dir> scifact against a throwaway PostgreSQL superuser cluster. The harness loads the data over HTTP, promotes it with reviewer keys, asks every query through oml_read and writes the result files listed below.

All suites

Quality

method queries nDCG@10 R@10 R@20 R@100 RR@10 S@1 S@5 zero
keyword-all 300 0.013 0.013 – 0.013 0.013 0.013 0.013 0.987
keyword-any 300 0.515 0.632 – 0.781 0.488 0.403 0.603 0.000
bm25 300 0.685 0.822 0.866 0.928 0.650 0.540 0.783 0.000
bm25-trigram 100 (sampled) 0.611 0.749 – 0.906 0.578 0.480 0.710 0.000
bm25-postings 300 0.685 0.822 0.866 0.928 0.650 0.540 0.783 0.000
sparse 300 0.710 0.826 0.856 0.931 0.681 0.613 0.780 0.000
bm25-sparse 300 0.730 0.853 0.877 0.957 0.697 0.610 0.800 0.000
bm25-sparse-rrf 300 0.731 0.852 0.882 0.954 0.701 0.620 0.797 0.000
bm25-k10 300 0.685 0.822 – 0.822 0.650 0.540 0.783 0.000
bm25-rerank-d20 100 (sampled) 0.699 0.805 – 0.925 0.676 0.580 0.800 0.000
bm25-sampled 100 (sampled) 0.696 0.800 – 0.925 0.671 0.580 0.780 0.000
bm25-trigram-k10 100 (sampled) 0.612 0.754 – 0.754 0.580 0.480 0.720 0.000
keyword-any-k10 300 0.515 0.632 – 0.632 0.488 0.403 0.603 0.000
reference-bm25s-k1-0.9-b-0.4 300 0.676 0.801 – 0.913 0.641 0.547 0.757 0.000
reference-bm25s-k1-1.2-b-0.75 300 0.679 0.808 – 0.913 0.643 0.543 0.753 0.000

Reference: BEIR BM25 (Anserini, k1 0.9, b 0.4, title and text fields), ndcg@10 0.665 (source).

Chart: nDCG@10 by pipeline

nDCG@10 on BEIR SciFact nDCG@10 on BEIR SciFact keyword-all 0.013 keyword-any 0.515 bm25 0.685 bm25-trigram 0.611 bm25-postings 0.685 sparse 0.710 bm25-sparse 0.730 bm25-sparse-rrf 0.731 bm25-k10 0.685 bm25-rerank-d20 0.699 bm25-sampled 0.696 bm25-trigram-k10 0.612 keyword-any-k10 0.515 reference-bm25s-k1-0.9-b-0.4 0.676 reference-bm25s-k1-1.2-b-0.75 0.679 published 0.665

Latency

Client wall time per recall, in milliseconds, nearest-rank percentiles.

method p50 p95 p99
keyword-all 629 672 701
keyword-any 815 1,075 1,203
bm25 86.1 116 130
bm25-trigram 3,707 5,454 6,284
bm25-postings 46.2 65.6 73.5
sparse 36.4 50.5 57.9
bm25-sparse 52.6 83.7 90.6
bm25-sparse-rrf 50.5 89.6 131
bm25-k10 70.5 98.5 109
bm25-rerank-d20 4,563 5,711 5,896
bm25-sampled 102 140 155
bm25-trigram-k10 3,664 5,334 6,358
keyword-any-k10 630 1,090 1,165
reference-bm25s-k1-0.9-b-0.4 offline offline offline
reference-bm25s-k1-1.2-b-0.75 offline offline offline
Chart: p95 latency by pipeline

p95 latency on BEIR SciFact p95 latency on BEIR SciFact keyword-all 672 ms keyword-any 1,075 ms bm25 116 ms bm25-trigram 5,454 ms bm25-postings 65.6 ms sparse 50.5 ms bm25-sparse 83.7 ms bm25-sparse-rrf 89.6 ms bm25-k10 98.5 ms bm25-rerank-d20 5,711 ms bm25-sampled 140 ms bm25-trigram-k10 5,334 ms keyword-any-k10 1,090 ms

What this result does not show

SciFact is scientific abstracts and claims. It does not predict recall over short chat notes.

Limits of this run

One abstract per memory as '. <text>' (BEIR indexes two fields); k 100, no token budget; English only.</p> <ul> <li>Offline bm25s over the same memories (Lucene BM25, English stop words, Snowball stems), not a served path;</li> <li>Asked with k 10, the size an agent asks, to measure latency at that k; recall@100 reads the shorter list.</li> <li>Served bm25 over the same seeded sample, on the same host in the same hour, for comparison with bm25-rerank-d20; the full-set bm25 row is the reference run.</li> <li>English only. bm25's list reranked by ms-marco-MiniLM-L6-v2 through candle on the CPU at depth 20 (the default became 10 after this run), with a cap of 8 and a 60 s deadline so no rerank was skipped; the host carried a load average of 9 to 29 from other work, and the LoCoMo run shared it with the NFCorpus run, so its latency is an upper bound (.</li> <li>English only. bm25-postings takes its candidates from the postings of the query forms it reads; when a query's readable postings pass POSTING_BUDGET (10,000) it leaves out the forms held by more than 10% of the readable rows, so a row holding only those is no candidate, and below the budget it reads every form and ranks as bm25.</li> </ul> <p>Sources: <code>eval/bench/results/beir-scifact/keyword-all.json</code>, <code>eval/bench/results/beir-scifact/keyword-any.json</code>, <code>eval/bench/results/beir-scifact/bm25.json</code>, <code>eval/bench/results/beir-scifact/bm25-trigram.json</code>, <code>eval/bench/results/beir-scifact/bm25-postings.json</code>, <code>eval/bench/results/beir-scifact/sparse.json</code>, <code>eval/bench/results/beir-scifact/bm25-sparse.json</code>, <code>eval/bench/results/beir-scifact/bm25-sparse-rrf.json</code>, <code>eval/bench/results/beir-scifact/bm25-k10.json</code>, <code>eval/bench/results/beir-scifact/bm25-rerank-d20.json</code>, <code>eval/bench/results/beir-scifact/bm25-sampled.json</code>, <code>eval/bench/results/beir-scifact/bm25-trigram-k10.json</code>, <code>eval/bench/results/beir-scifact/keyword-any-k10.json</code>, <code>eval/bench/results/beir-scifact/reference-bm25s-k1-0.9-b-0.4.json</code>, <code>eval/bench/results/beir-scifact/reference-bm25s-k1-1.2-b-0.75.json</code>.</p> </article> </div> <script>var tabs=__md_get("__tabs");if(Array.isArray(tabs))e:for(var set of document.querySelectorAll(".tabbed-set")){var labels=set.querySelector(".tabbed-labels");for(var tab of tabs)for(var label of labels.getElementsByTagName("label"))if(label.innerText.trim()===tab){var input=document.getElementById(label.htmlFor);input.checked=!0;continue e}}</script> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type="button" class="md-top md-icon" data-md-component="top" hidden> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class="md-footer"> <nav class="md-footer__inner md-grid" aria-label="Footer" > <a href="../recall-synthetic/" class="md-footer__link md-footer__link--prev" aria-label="Previous: Synthetic recall set"> <div class="md-footer__button md-icon"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg> </div> <div class="md-footer__title"> <span class="md-footer__direction"> Previous </span> <div class="md-ellipsis"> Synthetic recall set </div> </div> </a> <a href="../beir-nfcorpus/" class="md-footer__link md-footer__link--next" aria-label="Next: BEIR NFCorpus"> <div class="md-footer__title"> <span class="md-footer__direction"> Next </span> <div class="md-ellipsis"> BEIR NFCorpus </div> </div> <div class="md-footer__button md-icon"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11z"/></svg> </div> </a> </nav> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class="md-copyright"> <div class="md-copyright__highlight"> Code licensed <a href="https://github.com/e8kor/open-memory-lab/blob/main/LICENSE">Apache-2.0</a> · documentation licensed <a href="https://github.com/e8kor/open-memory-lab/blob/main/LICENSE-docs">CC BY 4.0</a> </div> Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"> Material for MkDocs </a> </div> </div> </div> </footer> </div> <div class="md-dialog" data-md-component="dialog"> <div class="md-dialog__inner md-typeset"></div> </div> <script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": ["navigation.tabs", "navigation.tabs.sticky", "navigation.indexes", "navigation.path", "navigation.footer", "navigation.top", "navigation.instant", "search.suggest", "search.highlight", "toc.follow", "content.tabs.link", "content.code.copy", "content.action.edit", "content.action.view"], "search": "../../assets/javascripts/workers/search.a7633da7.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script src="../../assets/javascripts/bundle.c21346df.min.js"></script> </body> </html>