Skip to content

feat(skills): external mounts get an 'external' provenance tier (#108032) - #128387

Merged
OutThisLife merged 5 commits into
mainfrom
bb/108032-external-skill-provenance
Oct 1, 2026
Merged

OutThisLife merged 5 commits into
mainfrom
bb/108032-external-skill-provenance

Conversation

@OutThisLife

@OutThisLife OutThisLife commented Sep 29, 2026 •

Copy link
Copy Markdown

What

Skills mounted from skills.external_dirs are externally authored, but every provenance surface flattened them into agent — the usage report, the dashboard's /api/skills set-based twin, the desktop catalog/detail UI, and the learning journey graph (where a used external mount rendered as a learned skill milestone).

This adds an external provenance tier at every site, without touching mutability.

Addresses #108032

Changes

  • tools/skill_usage.py — provenance() gains an external tier: an external-only skill (present under skills.external_dirs, absent from the local store — the local copy keeps winning on name clashes, mirroring is_agent_created's existing external exclusion) classifies as external. New _external_skill_names() scans external dirs once for the router's set-based twin.
  • hermes_cli/web_routers/skills.py — /api/skills classifies hub > bundled > external > agent.
  • agent/learning_graph.py — skills whose path resolves under an external dir are tagged source='external' (this is the shape the graph actually sees: the common symlinked SKILL.md into ~/.hermes/skills/, which the profile-root scan yields) and _has_learning_signal excludes them: a mount is configured, not learned, even when used.
  • Desktop — SkillInfo.provenance gains 'external'; the catalog labels mounts external (not local), and the detail pane shows an external skill as externally mounted while keeping it editable in place. The two provenance decisions are extracted into pure helpers (skill-provenance.ts) so they're testable directly.
  • hermes_cli/curator.py — curator usage reports and filters the external origin.

What this deliberately does NOT do

The issue also asks for skills.external_dirs to be read-only by default for foreground skill management. That half is not included because maintainers rejected it: commit 8c8fc6c1ec (PR #17512, which settled the then-open #4759 and #4381) removed exactly that default read-only gate — the refusal made agents create silent local duplicates, and the standing regression test at tests/tools/test_skill_manager_tool.py:784+ pins in-place external mutation succeeding. Re-imposing default-deny would resurrect a closed direction; the opt-in variant (external_dirs_readonly) is a separate live design question (see open PR #51412). This PR is the origin-labeling half only: the label says where a skill came from, and edit rights stay keyed to the standing April decision.

Prior art

Two earlier PRs addressed this issue and are both gone — #108040 (deleted) and #108767 (closed 2026-09-12 as its duplicate) — so this is a fresh implementation, not a salvage; no code was reused from either.

Testing

  • tests/tools/test_skill_provenance_external.py (new, 8 tests): tier classification (external-only → external, local stays agent, local copy wins, is_agent_created exclusion, names helper), the router twin through a real TestClient against the mounted app, and the journey graph (a used external skill produces no learning node via the symlinked-into-profile shape; a used local skill still does). 4 of 8 fail on origin/main sources, all pass with the change.
  • apps/desktop/src/app/capabilities/skills/skill-provenance.test.ts (new, 6 vitest cases): both helpers, including that bundled/hub stay non-editable and absent provenance keeps its legacy meaning.
  • Sibling suites green: test_skill_provenance.py, test_skill_usage.py, test_learning_graph.py, test_web_server_skills_profiles.py (38 tests).

@kiramakes

Copy link
Copy Markdown

🤖 Hermes Agent automated review: PR reviewed. Diff analyzed (424 lines changed). Check CI status and manual review recommended.

@whyyagswhy

Copy link
Copy Markdown

Independent verification on the PR head (2785b50): desktop provenance suite 6/6 green. Python provenance suite 7/8: test_used_local_skill_still_produces_learning_node fails under scripts/run_tests.sh but passes under direct pytest. Root cause is test isolation, not product code: the runner execs pytest under env -i, and the test only sees its fixture skill when ambient environment leaks in. If CI runs through run_tests.sh this test fails there too. Worth making the test hermetic before merge.

@arkheioncorp arkheioncorp left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

Clean implementation of the 'external' provenance tier. Key points:

  • Priority chain in provenance() is correct: hub > bundled > external > agent — local copy wins on name clashes via is_external_only()
  • External skills correctly excluded from learning graph (_has_learning_signal), while keeping in-place edit rights (isEditableProvenance)
  • Defensive import in learning_graph.py (try/except with fallback) is appropriate
  • Set-based twin in web router matches the per-skill provenance() semantics
  • Tests cover classification, learning graph exclusion, router twin, and the local-copies-external regression

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) tool/skills Skills system (list, view, manage) area/config Config system, migrations, profiles labels Sep 29, 2026
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

૮ >ﻌ< ა ci review

ran on 76dea6e — fix(desktop): fix lint in skill provenance files (hermes-age

debug info

CI timings

CI timings · View report · View job

Wall time 30m14s (no baseline yet).

@OutThisLife
OutThisLife force-pushed the bb/108032-external-skill-provenance branch from 2785b50 to e29848a Compare September 29, 2026 21:51
@OutThisLife

Copy link
Copy Markdown
Author

@whyyagswhy investigated on both the head you reviewed (2785b50) and the current one — reproduction says the test is not env-dependent, and the suite is green under the scrubbed runner on both:

  • scripts/run_tests.sh -j 1 tests/tools/test_skill_provenance_external.py → 8/8 green on 2785b50 and on the current head. The test was already hermetic: conftest's autouse _hermetic_environment fixture sandboxes HERMES_HOME (and every behavioral HERMES_* var) per test, and the fixture additionally sets its own HERMES_HOME + clears the external-dirs cache.
  • What you hit was the 'external' source loop-variable leak in build_skill_nodes, fixed in 71485f5 on this branch: at the reviewed head source = "external" overwrote the loop variable, so any skill yielded after the mount in one rglob scan inherited "external" and was dropped from the graph. Scan order depends on the tmpdir's on-disk layout, and env -i changes TMPDIR — so the failure correlated with the runner without being caused by the env.

To keep that order-dependence from ever hiding again, de01606 pins the hostile order directly: the new test_external_source_does_not_leak_to_later_skills_in_a_root patches Path.rglob so the external mount always scans before the local skill and asserts the local skill still classifies 'agent' — red at 2785b50 (my-local dropped entirely), green on the current head. Also removed the fixture's no-op monkeypatch.setattr("hermes_constants._hermes_home_cache", ...) — that attribute doesn't exist, so it reset nothing.

scripts/run_tests.sh -j 1 on the file: 9/9 green. New head: de01606.

@OutThisLife
OutThisLife enabled auto-merge (rebase) September 30, 2026 00:21
@alt-glitch alt-glitch added the needs-decision Awaiting maintainer decision before any implementation label Sep 30, 2026
)

Skills mounted from skills.external_dirs are externally authored, but every
provenance surface flattened them into 'agent': the usage report, the
dashboard's set-based twin, the desktop catalog/detail UI, and the learning
journey graph (where a used external mount rendered as a learned skill).

- tools/skill_usage: provenance() gains an 'external' tier (external-only:
  a local copy of the same name keeps winning, mirroring is_agent_created's
  external exclusion and _external_skill_names() scans external dirs once
  for the router's set-based twin.
- web_routers/skills: /api/skills classifies hub > bundled > external > agent.
- learning_graph: skills whose path resolves under an external dir are tagged
  source='external' and are never learning milestones — a mount is configured,
  not learned, even when used.
- desktop: SkillInfo.provenance gains 'external'; the catalog labels mounts
  'external' and the detail pane keeps them editable in place. Origin only,
  never mutability: read-only enforcement for external mounts was deliberately
  rejected by maintainers (commit 8c8fc6c / PR #17512 — a refusal just
  breeds silent local duplicates). Provenance decisions extracted into
  skill-provenance.ts pure helpers (testable without source-reading tests).
- curator: usage report/filter knows the 'external' origin.

The desktop edit logic in this commit is the natural reading of 8c8fc6c
(external dirs exist so the user can point Hermes at a vault they curate
elsewhere); if maintainers prefer the read-only half as well, it slots in
behind isEditableProvenance() in one place.
EOF
)
test_used_local_skill_still_produces_learning_node was reported as failing
under scripts/run_tests.sh (env -i) but passing under bare pytest, which read
as ambient-env leakage. Reproduction on both the reviewed head and the current
one shows the suite green under the scrubbed runner; the flake the reviewer hit
was the 'external' source loop-variable leak (fixed in 71485f5), whose
manifestation depends on rglob scan order — env -i changes TMPDIR, which
changes the tmpdir layout, which changes the order the mount and the local
skill are yielded in. That env-dependence was coincidental, not causal.

Pin the hostile order directly: patch Path.rglob so the external mount's
SKILL.md always scans before the local skill, and assert the local skill still
classifies 'agent' (red at the reviewed head, green now). Also drop the
no-op monkeypatch of hermes_constants._hermes_home_cache — no such attribute
exists, so it reset nothing and only implied a cache reset that never happened.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) needs-decision Awaiting maintainer decision before any implementation P3 Low — cosmetic, nice to have tool/skills Skills system (list, view, manage) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants