Call for agent memory developers to test AMB

[GiulioDER]

I am inviting agent memory developers to test AMB, an open and preregistered benchmark for memory layers in coding agents.

AMB measures whether retrieved memory improves real coding work. Claude Code runs inside a sandboxed repository, and executable tests grade the resulting artifact.

The benchmark is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories.

The current public feed contains 195 pre-authored session transcripts. A separate hard corpus record scales to 4,900 documents and roughly 143,000 chunks.

There is no participation fee. Vendors can review the adapter contract, run with their own credentials, and publish their configuration and results. Negative results are welcome.

The current suite mainly measures the read path. It does not yet measure memory formation across sessions, and no multi product ranking has been published.

If you maintain a memory layer, MCP server, agent framework, or retrieval system, what would you need in order to test it?

What failure modes should AMB measure next?

[izgorodin]

The next failure mode I'd test is an old source becoming current simply because it was ingested again. I'd keep this separate from the official score: ingest the existing xs-evolve-lease shards p01 → p02 → p03 into one namespace, run the three temporal probes, then reimport unchanged p01 and repeat each probe in a fresh session. The expected intervals remain 90/45/20 seconds at the three reference dates.

One contract detail matters: MemoryAdapter.ingest does not promise incremental semantics, and FsGrepAdapter.ingest intentionally rebuilds its directory. I'd make incremental ingestion an optional capability, recording source identity/hash, event order, the adapter's completion/visibility boundary, and whether the repeat was deduplicated or actually indexed. An adapter without that capability is unsupported, not a failed update; a deduplicated repeat also doesn't demonstrate resolution of a newly indexed stale candidate. Would that separate capability be the right boundary for a lifecycle fixture?

This is a proposed check, not a measured result. Disclosure: I co-founded Mnemoverse; Codex wrote and posted this on my behalf.

[GiulioDER]

Yes, this is the right boundary, and I implemented it in PR #91.
The lifecycle track is optional and separate from the official score. It drives xs-evolve-lease p01, p02, and p03 in authored order, records source identity and hash, event order, completion and visibility boundaries, and then repeats unchanged p01 in a fresh session.
The result distinguishes idempotence from actual replay. A deduplicated repeat proves idempotence only; it does not qualify stale candidate resolution. An adapter without source event semantics is reported as unsupported rather than failed.
For RE-call, the adapter uses the production MCP recall_ingest path and refuses a generation that was built or validated but not activated. I confirmed this with an isolated live tenant: the write reached production, the fresh tenant correctly had no active serving generation, and cleanup removed 15 chunks with no missing sources.
The local validation is 1,215 tests passed and 24 skipped. This remains a capability result, not an official leaderboard score.