How well does OmniGet's agent memory remember? Measured against Hindsight

OmniGet's built-in memory for AI agents and Hindsight answered the same 76 LoCoMo questions with the same model and the same judge. Accuracy at two context sizes and disk use.

Measured September 29, 2026 – views

Written by Antonio Leandro, who makes OmniGet. Every app is described from its own repository and releases.

Loop, OmniGet's mascot, and the Hindsight creature, a blue one-eyed figure with ringed arms, sit at one desk sorting glowing memory cards into two boxes.

Short answer

With 20 memories per answer OmniGet and Hindsight tie, 67.1% and 64.5%. With about 80, Hindsight wins, 71.1% against 63.2%. OmniGet's memory runs inside the desktop app with local models and a seventh of the disk.

Who wins what

  • Questions answered right, 20 memories per answer

    OmniGet

    OmniGet 67.1 % · Hindsight 64.5 %

  • Questions answered right, wide context

    Hindsight

    Hindsight 71.1 % · OmniGet 63.2 %

  • Disk used by the memory engine

    OmniGet

    OmniGet 316 MB · Hindsight 2,390 MB

Hindsight, by Vectorize, is one of the strongest open-source memory systems for AI agents: it turns conversations into facts, finds them again with four kinds of search and reports state-of-the-art scores on LongMemEval and LoCoMo. OmniGet's memory for its agents was rebuilt after studying it, so the fair question is how the two compare on the same data.

Both systems read the same LoCoMo conversation, 19 sessions of chat between two people, and then answered every second question outside the adversarial category, 76 in total. The same model, Claude Haiku through the Claude Code CLI, extracted the facts, wrote each answer from the memories the system returned, and judged the answer against the gold one. The only thing that changed between runs was the memory system.

Results

Questions answered right, 20 memories per answer Higher is better · %
  1. OmniGet 67.1 %
  2. Hindsight 64.5 %

Same model, answer prompt and judge for both; each tool picks its own 20 memories.

Questions answered right, wide context Higher is better · %
  1. Hindsight 71.1 %
  2. OmniGet 63.2 %

Hindsight's default recall (4,096 tokens, about 82 memories per answer) against OmniGet with 80.

Disk used by the memory engine Lower is better · MB
  1. OmniGet 316 MB
  2. Hindsight 2,390 MB

OmniGet: the three local model files and the ONNX runtime. Hindsight: its Python environment, the models it downloads on first start and the embedded PostgreSQL.

Seen enough? Get OmniGet for your system.

Download for Windows

Choose a file

All files and checksums

Free · no account · 64 MB · .exe

Windows may show a SmartScreen warning the first time. Here is what to do.

OmniGet is a desktop app

It runs on Windows, macOS and Linux. There is no Android or iOS version. Send this page to your computer:

Other platforms: Windows macOS Linux

The apps in this benchmark

  • OmniGet

    License:
    GPL-3.0
    Built with:
    Tauri (Rust + web UI)
    GitHub:
    14.4k stars · v0.10.1
    Version tested:
    OmniGet 0.10.1 with the agent-memory branch (not released yet) (September 21, 2026)
    • Its own extractors, torrents and direct files on top of yt-dlp.
    • An MCP server and AI coding agents in the same app.

  • Hindsight

    License:
    MIT
    Built with:
    Python server with PostgreSQL and pgvector (or embedded pg0)
    GitHub:
    60.5k stars · v1.2.1
    Version tested:
    hindsight-all 0.10.1 (embedded server, pg0) (September 21, 2026)
    • Published state-of-the-art results on LongMemEval and LoCoMo, reproduced by outside researchers.
    • Mature consolidation into observations and mental models, with clients for Python, Node.js, Go and a CLI.
    • Works with 25+ LLM providers and runs as a shared server for many agents and users.

What the numbers say

At the same context size OmniGet is slightly ahead, by two questions out of 76. By question type, with 20 memories: time questions 13 of 18 for OmniGet and 12 for Hindsight, multi-hop 8 of 15 against 7, single-hop 23 of 35 for both and open-domain 7 of 8 for both. The small edge sits where OmniGet dates relative events with a calendar at extraction time and walks its memory graph two steps instead of one. When each system may put many memories in the prompt, Hindsight pulls ahead, and its per-question answers show why: it keeps more small details per session, and those details matter once the prompt is wide. That gap is the next thing to close in OmniGet.

How much to trust a few points

One question is worth 1.3 points here, and two identical runs of the same system differed by up to three questions, because the answer and the judge are model calls. Treat differences under about four points as a tie. This is one LoCoMo conversation, not the whole benchmark, and the model is a small one; Hindsight's published scores use larger models and every conversation.

What it costs to run

OmniGet's memory is part of the desktop app: one SQLite file, a multilingual embedding model and a multilingual reranker that run on the computer, about 316 MB of disk in all. Hindsight needs a server with PostgreSQL, or its embedded mode with a Python environment, about 2.4 GB on this machine. Reading the 19 sessions took OmniGet 710 seconds and Hindsight 1,988, but OmniGet made four model calls at a time and Hindsight one, so that time is not a like-for-like speed result.

What this does not measure

Recall latency is left out: both ran debug or first-run builds on a shared machine, so the times would mislead. Nothing here covers Hindsight's mental models, directives or multi-user banks, or OmniGet's scopes, corrections and forgetting.

Which one should you pick?

  • Choose OmniGet if you want memory for the AI agents you already run on your computer, in any language, with no server to keep up, and you want to see and correct what they remember.
  • Choose Hindsight if you are building your own agent product, need a shared memory server for many users, or want the best accuracy when the prompt can hold a lot of context.

Where OmniGet still loses

We publish this on purpose. It is the list we work from to make OmniGet better.

  • Extract more small details per session: with about 80 memories in the prompt Hindsight answers 71.1% and OmniGet 63.2%.
  • Rank the tail better, so a wider context helps OmniGet's answers instead of diluting them.
  • Measure recall latency on a release build, alone on the machine.

Method

Run it yourself

node scripts/bench/agent-memory.mjs rebuilds the numbers from the per-question results in scripts/bench/agent-memory/. To measure again: the app's `cargo test -p omniget-core --lib assist::memory::bench::locomo_eval -- --ignored` (LOCOMO_K=20 and 80) and `python scripts/bench/agent-memory/hindsight_locomo.py` (default and TOPK=20), both on LoCoMo conv-26 with Claude Haiku through the Claude Code CLI.

Raw data (JSON)

Data licensed CC BY 4.0: reuse it freely with a link to this page.

Sources

Questions

Is OmniGet's memory more accurate than Hindsight?

On this test it was slightly ahead with 20 memories per answer (67.1% against 64.5%) and behind with a wide context (63.2% against 71.1%). Differences of a few points are within the noise of a 76-question run.

Does OmniGet's memory need a server or a database?

No. It lives in one SQLite file inside the app, and the search models run on the computer, so recalling a memory calls no AI model.

Which model did the test use?

Claude Haiku through the Claude Code CLI, for both systems, for extracting facts, answering and judging.

Other benchmarks

Try OmniGet 1.0.0

Free and open source, for Windows, macOS and Linux.

Download
Chat