Skip to content

Fixes from Hermes-side PRs worth carrying into hindsight-integrations/hermes #4662

Closed
Closed
Fixes from Hermes-side PRs worth carrying into hindsight-integrations/hermes#4662
Labels
integration:hermes

Description

@teknium1
teknium1
opened on Sep 23, 2026

Hi @nicoloboschi — follow-up to #4640. With the bundled provider removed from hermes-agent (NousResearch/hermes-agent#119888), we closed every open hermes-agent PR that patched plugins/memory/hindsight/ and redirected the authors here. Before closing, each PR was checked against hindsight-integrations/hermes at the catalog pin dc75038866a3966b5bf5c82ff57c7a758bb4070a (v0.10.1 era). This issue lists the ones whose fix still applies at that pin — the buggy code is present unchanged in your tree — so nothing is lost in the move. Feature proposals were not included (their authors were pointed here to open them directly).

Format: hermes-agent PR · author · one-line symptom · where in your tree (file:line at dc750388). Grouped by hotspot; several are duplicates of each other, which is itself useful signal about which bugs users hit most. Credit belongs to the linked authors.

Total: 117 closed PRs. Two hotspots account for a large share: embedded.py::_build_embedded_profile_env / _materialize_embedded_profile_env (profile-env rewrite drops or truncates HINDSIGHT_API_* keys → daemon restarts with the wrong embeddings/LLM config) and __init__.py::_tool_retain (the manual hindsight_retain tool ignores retain_async and blocks the turn).

__init__.py::_fetch_hindsight_api_version (2 PRs)

__init__.py::_check_api_supports_update_mode_append (2 PRs)

__init__.py::_get_loop (1 PR)

__init__.py::_load_config (1 PR)

__init__.py::_mint_document_id (1 PR)

__init__.py::is_available (2 PRs)

__init__.py::unavailable_reason (1 PR)

__init__.py::_new_embedded_client (1 PR)

__init__.py::_new_cloud_client (1 PR)

__init__.py::_get_client (3 PRs)

__init__.py::_run_hindsight_operation (2 PRs)

__init__.py::_ensure_writer (1 PR)

__init__.py::_writer_loop (1 PR)

__init__.py::_resolve_retain_target (2 PRs)

__init__.py::_apply_connection_settings (5 PRs)

__init__.py::_apply_retain_policy (1 PR)

__init__.py::_apply_recall_settings (3 PRs)

__init__.py::_daemon_start_worker (8 PRs)

__init__.py::system_prompt_block (1 PR)

__init__.py::_reflect (2 PRs)

__init__.py::_do_recall (5 PRs)

__init__.py::prefetch (1 PR)

__init__.py::queue_prefetch (3 PRs)

__init__.py::_build_turn_messages (3 PRs)

__init__.py::_make_turn_retain_job (3 PRs)

__init__.py::sync_turn (5 PRs)

__init__.py::_enqueue_retain (1 PR)

__init__.py::_tool_retain (10 PRs)

__init__.py::handle_tool_call (1 PR)

__init__.py::on_session_switch (2 PRs)

__init__.py::shutdown (6 PRs)

__init__.py::register (1 PR)

config_schema.py::CONFIG_SCHEMA (3 PRs)

embedded.py::_check_local_runtime (2 PRs)

embedded.py::_ensure_local_runtime (1 PR)

embedded.py::_build_embedded_profile_env (17 PRs)

embedded.py::_secure_write_profile_env (1 PR)

embedded.py::_materialize_embedded_profile_env (5 PRs)

settings.py::_OBSERVATION_SCOPE_KEYWORDS (module constants) (3 PRs)

settings.py::_normalize_observation_scopes (1 PR)

setup.py::run_setup (2 PRs)

Also seen in hermes-agent PRs that stay open for an unrelated core-side change

These PRs remain open on hermes-agent because they also carry a generic core fix; their Hindsight half is dead there but the same bug exists at the pin:

Notes

Activity

  1. UniGood commented on Sep 23, 2026

    @UniGood
    UniGood
    on Sep 23, 2026

    Taking the __init__.py::prefetch / queue_prefetch entries (NousResearch/hermes-agent#42232).

    I'll port both halves to hindsight-integrations/hermes and open a PR here shortly:

    1. prefetch_join_timeout config key (guarded float parser, schema, README) — the joins at __init__.py:1235 and :1525 are still hardcoded to 3.0s while a typical recall takes 1.5-4.4s, so prefetch often returns empty.
    2. _prefetch_generation guard in queue_prefetch — closes the race where a timed-out switch join lets an old-session worker write stale recall into the new session's cache (same class as #117244 / #70019 / #64745).

    Will self-review against the pin before sending. Shout if someone is already on this and I'll stand down.

  2. qxxaa
    added a commit that references this issue on Sep 23, 2026

    fix(hermes): retain conversation turns as a flat message array

    2970635
  3. yingliang-zhang
    added a commit that references this issue on Sep 23, 2026

    fix(hermes): pass retain_async=True to aretain_batch in tool handler

    4d84463
  4. qxxaa
    added a commit that references this issue on Sep 23, 2026

    fix(hermes): retain conversation turns as a flat message array

    d69136f

  5. yaozhen commented on Sep 23, 2026

    @yaozhen
    yaozhen
    on Sep 23, 2026

    Adding measurements from a live install, plus two things that are not yet in your list. Environment: hermes-agent v0.21.3 (editable install), hindsight-embed 0.9.2, local_embedded, profile hermes, macOS 15.7.7, LLM openai provider → OpenAI-compatible base URL (DeepSeek V4.1 Flash).

    1. Quantified impact of the drift-compare restart (adds to #83076 / #104324 / #107324 / #118858)

    All four of those reports are the same root cause and it is still live at 0.9.2:

    • === Config changed, restarting daemon === in ~/.hermes/logs/hindsight-embed.log: 18 in a single day (up to 21/day) while the daemon was in active use — each one stopping a healthy, running daemon and paying a cold start.
    • Cold start measured at ~24.9s (embedded PostgreSQL + model load). On a memory-constrained host this is also the window where every retain/recall queues behind the boot.
    • Downstream effect: retain ops are judged "recall-visible" by the client through _wait_for_retains_drained() with a 10.0s budget (prefetch_retain_drain_timeout default). Measured single retain op duration from the daemon's own *_RETAIN COMPLETE: … in Ns lines: median 13.4–14.0s, 66% over 10s (4 days, one install) — i.e. the budget is structurally smaller than the work, and the warning Prefetch: server retain visibility timed out after 10.0s; dropping N unresolved op(s) fires on ~74% of turns. Restarts make this worse but are not the main term (30% of those timeouts land within 180s of a restart, vs 3.75% of wall-clock covered by those windows).
    • Reproduce the false drift in two lines, no daemon start needed:
    cfg = _load_config()
_load_simple_env(_embedded_profile_env_path(cfg)) != _build_embedded_profile_env(cfg)   # always True
# disk: template + 7 keys (incl. HINDSIGHT_API_PORT) ; build: 6 keys

    2. Not in the list: per-operation LLM keys are unreachable in the embedded path

    This matters because several of the listed fixes are "profile env drops keys", but fixing preservation alone does not make the per-op LLM knobs usable:

    • hindsight_api supports them (config.py: HINDSIGHT_API_RETAIN_LLM_MODEL / _REASONING_EFFORT / _EXTRA_BODY, parsed with json.loads).
    • In the embedded path the plugin forwards only six keys as kwargs (profile, llm_provider, llm_api_key, llm_model, idle_timeout, llm_base_url), so anything in config.json outside those is not forwarded.
    • Writing them into ~/.hindsight/profiles/<p>.env does not survive: the file is rewritten from the bundled template on every daemon start (the same rewrite as above).
    • What does work: ambient environment. daemon_embed_manager builds the child env as env = os.environ.copy() and only then merges the profile config (daemon_embed_manager.py:725 and the HINDSIGHT_* filter below it), so a HINDSIGHT_* key in the Hermes .env (loaded into the gateway process, inherited by the daemon) reaches the daemon and survives the rewrite. Verified on the running process (ps -Eww <hindsight-api pid> shows HINDSIGHT_* keys that exist only in the Hermes .env).

    Request: when porting the env-preservation fixes, please also give the integration a supported config path for the per-operation LLM keys (model / reasoning_effort / extra_body per operation) instead of leaving ambient env as the only route.

    3. Not in the list: the retain extraction spends most of its time on hidden reasoning

    Same install, same prompt, no truncation, model deepseek-v4.1-flash served through an OpenAI-compatible aggregator:

    variant wall completion tokens of which reasoning facts extracted
    default (no param sent) 12.0s 2566 2328 8
    reasoning_effort: "none" 10.2s 2100 1847 8
    reasoning_effort: "low" 18.9s 4000 (cap hit) 4000 —
    thinking: {"type": "disabled"} 1.8s 342 0 8

    Two takeaways for the integration: (a) reasoning_effort alone does not disable thinking on this OpenAI-compatible backend (DeepSeek's own docs: none disables, low/high/max all enable, default high), so a "set effort to none" fix is not enough — the thinking field (or an equivalent extra_body) is what works; (b) with thinking off, a retain op drops from ~14s to ~2s, which makes the 10s visibility budget meaningful again. Worth considering whether the default prefetch_retains_drain_timeout: 10.0 should stay coupled to a model-dependent op duration at all, or whether the wait should be opt-in.

    Happy to re-measure against hindsight-integrations/hermes once the port lands.

  6. Driskol commented on Sep 23, 2026

    @Driskol
    Driskol
    on Sep 23, 2026

    Opened #4694 for the _daemon_start_worker / _build_embedded_profile_env entry of this list: the drift check in _start_daemon now compares only the keys _build_embedded_profile_env produces, so a foreign key living in the shared profile env (HINDSIGHT_API_PORT, written by the daemon itself) no longer rewrites the file and stops a healthy daemon. Reproduced with the plugin's own fixtures (6 keys on disk vs 5 built: old compare True, new check False) and uv run pytest tests 30/30 green.

    On the _reflect entries (#62205, #115629): the tag-filter forwarding is the one that bites in practice. The eager version of it — forward the configured recall_tags / recall_tags_match to areflect, same as _recall already does — ran in production on a bank with an any_strict speciality filter for a while, and the symptom it removes is the whole-bank scan: 105k-156k tokens retrieved against a 100k context, split synthesis, 300s wall-clock timeout and an empty reply; with the filter, 55s and a real 11k-character synthesis. If you would rather not change the default, the opt-in reflect_tags variant from NousResearch/hermes-agent#115501 (closed with the handoff) is on the table too — the config normalization it uses comes from the retained-tag path. Happy to port either one if that helps; I did not want to open a competing PR against a _reflect entry that is already on this list.

  7. balloon5 commented on Sep 25, 2026

    @balloon5
    balloon5
    on Sep 25, 2026

    Windows data point for the config_schema.py::CONFIG_SCHEMA entry (carried over from hermes-agent #84572 / #83644 / #84586) — the confirmations so far are Linux-only:

    • Windows 11 (build 26200), shipped desktop build · Hermes v0.21.3 (2026.9.14) / upstream bc655bfb · still on the bundled provider (pre-migration): <profile>/hindsight/config.json holds "mode": "local_embedded", the agent log shows Hindsight initialized: mode=local_embedded, the embedded daemon serves 127.0.0.1:9177 and recall/retain work — while the panel renders Mode = Cloud with the cloud API URL.
    • The same config read through the legacy surface (provider.config_schema()) returns local_embedded correctly, so the stored value is intact; only the declared-schema path coerces it.
    • Not fixed by the migration: @azaryc2m reports the same with the catalog plugin as of 2026-09-25.
    • The only local mitigation is "never touch the panel" — any panel save flattens local_embedded to cloud.

    Checked the default branch today: hindsight-integrations/hermes/config_schema.py (f38d53b0ad) still declares only cloud / local_external at lines 22–25, so the five-line fix from #83644 still applies as-is. Has anyone picked this one up? If a PR is welcome, I'm happy to test a build on Windows 11 + v0.21.3 and report back.

  8. woshicby commented on Sep 26, 2026

    @woshicby
    woshicby
    on Sep 26, 2026

    Taking the __init__.py::unavailable_reason entry (NousResearch/hermes-agent#86078) — ported it into hindsight-integrations/hermes: #4808.

    At the current pin, cloud mode still returned "" from unavailable_reason(), so the host's "provider unavailable" warning (which already explains that a gateway/systemd unit does not inherit ~/.hermes/.env) never named the variables to set.

    The port keeps the local branches untouched — local_embedded keeps the hindsight-all install hint, local_external stays hint-free because it is available without cloud credentials — and adds the cloud case with the same three inputs is_available() gates on (apiKey, api_url, HINDSIGHT_API_URL).

    Patch + tests: 7 new cases, uv run pytest tests -q → 33 passed.

    Happy to fold this into an existing PR if someone is already on it.

  9. nicoloboschi
    added
    integration:hermes
    on Sep 28, 2026

  10. slyvixenops commented on Sep 29, 2026

    @slyvixenops
    slyvixenops
    on Sep 29, 2026

    Two additions to the carried-fix hotspots from running this integration on a PM-managed Hermes install (py3.14 generation env):

    1. _find_api_command uv fallback — silent ABI death on every auto-start. When the fallback resolves the hindsight-api console script from the uv archive cache, that binary's interpreter is a different Python minor than the generation env whose site-packages the spawn inherits (pydantic_core and other compiled deps load from the gen env). Result: ModuleNotFoundError: No module named 'pydantic_core._pydantic_core' on every auto-start, each attempt burning the full start-timeout window before failing. The fix shape that holds up in production here: resolve the command from the imported hindsight_embed package's own environment — its sibling bin/hindsight-api, or <that env's python> -m hindsight_api.main — so the spawned daemon is independent of the parent's sys.path/PYTHONPATH entirely.

    2. Profile-env rewrites have two writers with different formats — user lines silently dropped. The Hermes-side plugin materializes a minimal KEY=VALUE profile env, while the embed manager's create_profile renders the full documented template (render_config) — and each rewrite drops every active line the other side didn't include. Any user-added profile-env setting (we lost a concurrency-tuning line this way) survives only until the next rewrite by whichever writer runs. Suggest folding into the 17-PR profile-env-rewrite hotspot: whichever component owns the file long-term, the other's format must merge rather than replace, or the file needs a "user lines" section both writers preserve.

  11. fleet2jeep commented on Sep 29, 2026

    @fleet2jeep
    fleet2jeep
    on Sep 29, 2026

    Dropping the O_TRUNC write-path case here per the scoping in #4694 — the compare fix deliberately leaves this half alone, so pinning it red while it's still unfixed.

    The case (from the production host that ran the compare fix as a local patch since 2026-08-31): when a genuine drift rewrite does fire — model changed, base URL rotated, whatever the governed compare correctly flags — _secure_write_profile_env opens the file O_TRUNC and writes only the keys _build_embedded_profile_env produces. Every operator-owned or daemon-registered key that isn't in that build disappears in one write. On the live host this ate HF_HUB_OFFLINE=1 and HINDSIGHT_EMBEDDINGS_LOCAL_FORCE_CPU=true — a silently changed embedder behavior with nothing in the logs to say so.

    The red test (runs green against fix/profile-env-drift-compare as a failing assertion, i.e. it documents today's behavior):

    def test_operator_owned_keys_do_not_survive_a_genuine_drift_rewrite(hermes_env):
    _materialize()
    path = embedded._embedded_profile_env_path(dict(CONFIG))
    with path.open("a", encoding="utf-8") as fh:
        fh.write("HF_HUB_OFFLINE=1\n")
        fh.write("HINDSIGHT_EMBEDDINGS_LOCAL_FORCE_CPU=true\n")
    _materialize({**CONFIG, "llm_model": "another-model"})  # real drift triggers the rewrite
    on_disk = embedded._load_simple_env(path)
    assert "HF_HUB_OFFLINE" not in on_disk  # RED today: O_TRUNC dropped it

    The flip side: when the compare fix lands, rewrite frequency drops to "only on genuine drift," which makes each rewrite more costly to get wrong — the rare rewrite that silently drops an operator's HF_HUB_OFFLINE is arguably harder to notice than the old every-start churn everyone learned to watch for. So the two halves compound: the compare fix makes restarts rare, and this half decides whether those rare rewrites are lossy.

    Happy to rework it into a merge-preserving formulation (e.g. _materialize seeding foreign keys first, rewrite preserving keys outside the governed set) if that matches where you want to take it. Full context and the other three production cases: #4694 (comment)

  12. Driskol
    added a commit that references this issue on Sep 29, 2026

    test(hermes): port the profile-env production cases from vectorize-io…

    66d31df
  13. yingliang-zhang
    added 3 commits that reference this issue on Sep 30, 2026

    fix(hermes): pass retain_async=True to aretain_batch in tool handler

    ac0d77b

    fix(hermes): fence prefetch publication to the owning session generation

    5a5096f

    fix(hermes): bind retain work to enqueue-time session identity

    03bb459

  14. 8 remaining items

  15. dudu7615 commented on Oct 6, 2026

    @dudu7615
    dudu7615
    on Oct 6, 2026

    Hi @carlosrve , thanks for keeping this tracked and for making the session-end boundary explicit in #5218 — that is what prompted me to measure it.

    Here is the deterministic, provider-level measurement you asked for — four cases at retain_every_n_turns=8 with two turns buffered below cadence, driven through the recording client (no live server, no private bank):

    case path base @ cc806c4d6 + on_session_end override
    1 on_session_end alone (no switch, no shutdown) tail dropped 1 retain
    2 on_session_switch 1 retain (old doc id) 1 retain
    3 shutdown alone tail dropped tail dropped — still open, tracked in #5218
    4 on_session_end → on_session_switch (CLI /new order) — exactly 1 retain, not 2

    Case 1 is the session-end gap you flagged in #5218: with no override the base hook is a no-op and shutdown() only drains the writer queue, so the buffered tail is lost. Case 4 is the idempotence guard — clearing the buffer on flush means the following switch has nothing to re-send. RED contrast: commenting out only self._session_turns.clear() makes case 4 fail with assert 2 == 1, both retains carrying byte-identical payloads (the #41911 duplicate).

    Case 3 I record as not fixed by this: shutdown() never touches the buffer, so it belongs to the durable-outbox work tracked in #5218, not to the on_session_end override — asserting the loss keeps that boundary explicit.

    Branch dudu7615/hindsight fix/hermes-session-end-boundary-evidence @ e46e92699; tests in tests/test_session_end_boundary.py. Also independently confirmed on a live self-hosted install (real daemon, scratch bank): end→switch yields the tail once, no duplicate document. Happy to run more cases or adjust the assertions if that would help.

  16. carlosrve commented on Oct 7, 2026

    @carlosrve
    carlosrve
    on Oct 7, 2026
    Contributor

    Thanks @dudu7615 — the four-case measurement and RED contrast make the session-end boundary much clearer. Your distinction between the end-hook fix and the shutdown-only tail is correct.

    Our reference fork now covers both boundaries in the merged outbox feature. At the validated composed revision, test_retain_outbox.py includes shutdown below cadence, session-end without switch, repeated end followed by switch/shutdown in append and legacy modes, failed staging preserving the buffer, and API-outage recovery by a new provider without a new turn. The pinned real-Hermes callback probe and scripted-model API/Postgres tests also passed in the final CI run.

    These are deterministic reference tests, not a claim that upstream main has the fix or that transport delivery is exactly once after an ambiguous acknowledgement. API acceptance is distinct from later extraction completion. #5218 remains the focused durability report; your independent evidence is valuable for its acceptance criteria. No additional measurements are needed from you for the four cases above.

  17. yingliang-zhang commented on Oct 8, 2026

    @yingliang-zhang
    yingliang-zhang
    on Oct 8, 2026
    Contributor

    Verification: are the fixes from the closed hermes stack (#4671–#4676) on main?

    Ran the regression tests from those PRs against main (fb11ddf) in hindsight-integrations/hermes (provider source from main, fixtures from tests/conftest.py): 7 passed, 11 failed.

    Closed PR Fix On main? Test evidence
    #4674 client lifecycle leaf lock yes 4/4 test_client_lifecycle.py green; _client_lock present in __init__.py
    #4671 forward the configured retain_async in _tool_retain no both parametrized cases fail with KeyError: 'retain_async'
    #4672 prefetch publication generation fence no a stale worker publishes the old session's recall into the new session's slot; an older worker clobbers a newer result; a late publish lands after shutdown()
    #4673 build the retain item at enqueue time no a queued retain picks up new-tag after a mid-flight session switch (['new-tag', 'session:session-1'] instead of ['old-tag', 'session:session-1'])
    #4675 prefetch spawn owner gate no a late prefetch for a superseded session still publishes Memory 1/2 into the child's slot
    #4676 untrusted recall framing (bounded whitelisted JSON) no raw bullets including </memory-context><forged>… and a code-fence system block reach the prompt verbatim

    Notes:

  18. carlosrve commented on Oct 9, 2026

    @carlosrve
    carlosrve
    on Oct 9, 2026
    Contributor

    Independently reproduced the manual-retain policy and stale-prefetch publication cases against our pinned reference integration (e211b2b00). They are now separated in maintained-fork reference PRs #7 and #8; credit to @yingliang-zhang for #4671/#4672/#4675 and the independent regression report above. No new external PR is being submitted.

    The prefetch tests explicitly release blocked workers after switch, end, shutdown and a newer empty response; owner admission is also checked before spawning. A bounded join is not treated as worker cancellation. Manual retain tests record the forwarded retain_async value for both settings. The temporary combination passed 81 deterministic provider tests, including the pinned real-Hermes callback probes. API clients are mocked for these tests; acceptance is not extraction completion.

    The separate framing report is now #5424. These references establish a reproducible patch/test path, not that upstream main contains the fixes or that a production instance was updated.

  19. yaozhen commented on Oct 9, 2026

    @yaozhen
    yaozhen
    on Oct 9, 2026

    Follow-up from the install I reported on 2026-09-23 (same host, still local_embedded): the drift-compare fix is merged and correct, but it is not reachable from the Hermes plugin catalog yet — catalog installs are still getting the pre-fix tree.

    The fix (confirmed merged)
    fd70557c6 — fix(hermes,embed): local_embedded daemon start, restarts and Windows launch (#5360), 2026-10-06 — introduces the owned-key set (_PLUGIN_PROFILE_ENV_KEYS) and compares on those keys only, i.e. the #5222 root cause. Its patch adds the key set, and the same code path is present in the v0.10.3 tree (2026-10-08) and on main.

    Why it doesn't arrive
    The catalog pin for this provider is release(hermes): v1.2.1 = d56c4acd, 2026-09-30 — 257 commits behind v0.10.3. hindsight-integrations/hermes/plugin.yaml on main still declares version: 1.2.1, the same string as that release commit, so nothing downstream can distinguish the fixed tree from the unfixed one. Verified on a live install: hermes plugins browse shows 1.2.1 @ d56c4acd, and hermes plugins update re-installs it — grep -c _PLUGIN_PROFILE_ENV_KEYS in the installed tree = 0.

    Measured today (2026-10-09, after updating the plugin v1.0.1 → v1.2.1 and restarting the gateway)

    • === Config changed, restarting daemon === keeps firing: 217 cumulative, 20 of them on 2026-10-08 (a busy session day; 3–6/day on quiet days). Each one SIGTERMs a healthy, running daemon.
    • Both writers caught on the same file within 60 s: 11:31 = the plugin's write (6 keys, 295 B), 11:32:23 = the daemon's create_profile template again (701 lines, 46,321 B, containing HINDSIGHT_API_PORT=9177) — that port line is exactly what the full-dict compare reads as drift.
    • Not collateral damage: --idle-timeout 1800 survives, and the local embedder/reranker (BAAI/bge-small-zh-v1.5, dim 512, cpu; BAAI/bge-reranker-v2-m3) load cleanly on each cold start (11:31:17 → 11:31:51). The profile env keeps the LLM provider/key/base-url/model, so no keyless boot. The cost is availability (~16 s per restart), not correctness.

    Ask: is a new release(hermes) (or a bump to 1.2.2) planned so catalog pins can carry fd70557c6? Otherwise every catalog consumer stays on the unfixed tree until the pin is re-pointed manually — happy to re-verify on this host once a pin carrying it is published. /cc @nicoloboschi

  20. Slaventuss commented on Oct 11, 2026

    @Slaventuss
    Slaventuss
    on Oct 11, 2026

    Still live at 1.2.1 / main — the _tool_retain entry on this list is a user-visible failure, not a cosmetic gap.

    Two PRs already target exactly this (#4671, #4670) and both were closed without merge, while hindsight-integrations/hermes/__init__.py at main still ships the synchronous call:

    • _tool_retain → self._retain_batch(item, bank_id=self._bank_id) — retain_async never forwarded
    • _retain_batch → kwargs = {k: v for k, v in kwargs.items() if v is not None} — drops the None, so the client's own default (aretain_batch(..., retain_async: bool = False)) applies → SYNC
    • the turn/auto path (_make_turn_retain_job) always passes retain_async=self._retain_async, which is why only agent-chosen retains break

    Measured on a live install (Hermes profile + plugin 1.2.1, daemon healthy, idle queue):

    • tool path with retain_async=True: 0.46–0.52 s, returns an operation_id
    • the same call through the current code path: 3.5 s / 11.2 s — the call time is the server-side queue wait, so it grows with the queue and the 120 s HINDSIGHT_TIMEOUT is the cliff

    Consequence for users: past 120 s the plugin's _run_sync raises concurrent.futures.TimeoutError, whose str() is empty, and the tool handler renders it as the opaque user-visible Failed to store memory: with no reason. ~18 episodes logged on one install over two months, while /health, parked-ops and the memory-LLM probes all stayed green — the failure is purely client-side, so no server-side monitoring can see it.

    Minimal carry-over (this is #4671's diff; it still applies cleanly at main):

             logger.debug("Tool hindsight_retain: bank=%s, ...", ...)
-        self._retain_batch(item, bank_id=self._bank_id)
+        resp = self._retain_batch(item, bank_id=self._bank_id, retain_async=self._retain_async)
+        if self._retain_async:
+            self._track_retain_ops(resp, self._bank_id)
         logger.debug("Tool hindsight_retain: success")

    plus a belt-and-braces default in _retain_batch before the kwargs dict is built:

    if retain_async is None:
    retain_async = self._retain_async

    Happy to test a patch branch against a real install (postgres, 17k-fact bank, Hermes 0.21.4, local_external mode) if that helps. Filed as a follow-up to the closed PRs rather than a new issue, since this list is the right home for it. Unrelated to #5486 (env-compare restart loop), which I filed separately.

  21. hugetoucan commented on Oct 11, 2026

    @hugetoucan
    hugetoucan
    on Oct 11, 2026

    Independent live-deployment confirmation that the prefetch identity / session-race fixes tracked here (from #4672, #4675, #4664) still apply at current main.

    Environment. Windows host; hindsight-integrations/hermes plugin (local copy reports 1.2.1); Hindsight daemon 0.10.3; recall_sync=false, prefetch_waits_for_retain=true. Measured recall latency on this box: p50 39 s, max 116 s (266 completed recalls).

    Checked against current main today. In hindsight-integrations/hermes/__init__.py (main, still reporting version: 1.2.1): queue_prefetch() still spawns a thread per call without checking whether the previous worker is alive, still publishes into the shared _prefetch_result unconditionally, and on_session_switch() still only _join_prefetch(3.0) before clearing the slot. No generation / epoch / slot-owner guard is present, so the ported fixes are still not in the tree.

    Reproduction (no live server, no LLM, no network: the installed methods are AST-extracted and driven with an in-memory fake thread that completes only when instructed):

    1. Two work items queued for consecutive turns, A then B. B completes and publishes MEMORY_FOR:newer-turn-B; A completes afterwards and overwrites the slot with MEMORY_FOR:older-turn-A; the next prefetch() consumes A — i.e. the stale result wins.
    2. Session switch S1→S2 clears the slot, but S1's still-running worker completes later and publishes; the next consume in S2 receives S1's result.

    Why the bounded joins do not cover it. The worker budget is up to 10 s in _wait_for_retains_drained() plus the recall itself, while both joins are 3.0 s. Any recall slower than the join outlives a session switch, and with overlapping spawns an older completion can overwrite a newer one.

    Supporting live signal (approximate, derived from daemon log finish time + duration across two rotated log files): 67 overlapping adjacent complete recall blocks out of 511 — e.g. two recalls finishing 15 s apart with ~41.9 s of overlap.

    Would maintainers consider carrying the fence into the plugin — publication fenced to the owning session generation, plus the spawn gate on the slot owner? A bare if thread.is_alive(): return is not sufficient (it drops the pending query). Happy to re-submit or re-open either PR, or to attach the reproduction harness if that would help.

  22. teknium1
    added 2 commits that reference this issue on Oct 12, 2026

    plugin-catalog: re-pin hindsight to v0.10.3 (embedded-daemon fix)

    5ed4451

    plugin-catalog: re-pin hindsight to v0.10.3 (embedded-daemon fix)

    bcec4b5

  23. teknium1 commented on Oct 12, 2026

    @teknium1
    teknium1
    on Oct 12, 2026
    Author

    Heads-up for the catalog side of this: the Hermes plugin catalog now pins hindsight at eb6df499 (v0.10.3), which includes fd70557c6 (local_embedded daemon start, restarts and Windows launch). Landed in NousResearch/hermes-agent#137085 (bcec4b56d9). Thanks @yaozhen for flagging that catalog installs were still getting the pre-fix tree.

    Existing installs pick it up with hermes plugins update hindsight. Clients cache the catalog for a few hours, so run hermes update first, or delete ~/.hermes/cache/plugin-catalog.json, if the update isn't offered yet.

  24. nicoloboschi
    added a commit that references this issue on Oct 12, 2026

    fix(hermes): batch of small plugin bugs from #4662 (#5534)

    37f0cd1

  25. carlosrve commented on Oct 12, 2026

    @carlosrve
    carlosrve
    on Oct 12, 2026
    Contributor

    Follow-up after #5534: independently checked upstream f843f173bb41f001e322cc83c917371639e47fbf with SDK/embed 0.10.3, simulated clients only. The upstream provider suite passes 139 tests. From our directed reference regressions, both manual-retain retain_async settings now pass; synchronous current-query recall and the three late-completion cases across switch/shutdown/newer-empty-query also pass. These are deterministic provider measurements, not live latency/extraction results.

    One narrower ownership boundary still fails: after switching from session S1 to S2, queue_prefetch(..., session_id=S1) admits a new worker. The new generation fence blocks an already-running old worker, but queue_prefetch still ignores its supplied session owner, so a late queued callback can create a new current-generation request with S1's query. The existing public reference regression reproduces this without a server or LLM (test_superseded_owner_cannot_spawn_prefetch). Our narrowed run: 6 passed / 1 failed, with the separate session-end and outbox-flush fixture cases excluded because upstream does not expose those same callbacks/helpers.

    This is a remaining admission/owner guard, not a request to restore the already-merged generation fix or reopen an external PR. Thanks for carrying the manual policy and publication fence into upstream. The catalog availability update is also verified independently; it does not by itself establish that every newer main change is already in a catalog install.

  26. Xipong commented on Oct 15, 2026

    @Xipong
    Xipong
    on Oct 15, 2026

    Follow-up for my transferred NousResearch/hermes-agent#117513 contribution: opened focused bug #5583 after checking standalone plugin 1.2.2 at f843f17 and current main d5b1d42. A no-server probe still confirms automatic recall, human-cache consumption, queued recall and retention-buffer admission for runtime-shaped inputs. The new report separates Hindsight-owned policy from the missing supported host provenance contract and explicitly requests compatibility with unmodified supported Hermes versions, not a hard import of the unpublished local agent.turn_origin module. It includes the executable reproducer, current source links and human → notification → human acceptance criteria.

  27. nicoloboschi commented on Oct 15, 2026

    @nicoloboschi
    nicoloboschi
    on Oct 15, 2026
    Collaborator

    Thanks for the careful handoff, this list was very useful. Closing it now that the bulk is fixed and released in hermes plugin v1.2.2:

    • fix(hermes,embed): local_embedded daemon start, restarts and Windows launch #5360: the profile-env hotspot. The drift check now only compares the keys the plugin writes, so the daemon no longer restarts on every new Hermes process. The rewrite also keeps keys it doesn't own (port, operator settings), so HINDSIGHT_API_* keys you add to the profile env now survive. @yaozhen, that covers the per-operation LLM keys (HINDSIGHT_API_RETAIN_LLM_MODEL etc.) too.
    • fix(hermes): batch of small plugin bugs from #4662 #5534: the batch of small bugs. The hindsight_retain tool honours retain_async, stale prefetch can't land in a newer session, recall_tags accepts a comma-separated string, bad numeric settings fall back to defaults, the reconnect closes the old client, shutdown unregisters its atexit hook, and more.

    What was still open on main is now split into self-contained issues:

    If you hit something from the list that isn't covered by the above, please open a new issue.

  28. nicoloboschi
    closed this as completedon Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    integration:hermes

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions