Skip to content

feat: remote catalog refresh with caching (#76) - #231

Open
albin-george-kurian wants to merge 1 commit into
OpenAgentHQ:mainfrom
albin-george-kurian:feature/remote-registry-catalog
Open

feat: remote catalog refresh with caching (#76)#231
albin-george-kurian wants to merge 1 commit into
OpenAgentHQ:mainfrom
albin-george-kurian:feature/remote-registry-catalog

Conversation

@albin-george-kurian

Copy link
Copy Markdown
Contributor

Summary

Closes #76.

The catalog was effectively bundled-only: RemoteRegistry existed but was never
constructed anywhere, and registry_url was parsed, env-mapped and printed by
modeldock config show while being read nowhere in src/. Even when constructed
manually it did not do its job — it never cached, and a successful fetch replaced
the catalog instead of extending it, while search() and recommend() delegated
straight to the bundled fallback, so a model that existed only in the remote catalog
could never be found. That is the exact case the feature exists for.

This rebuilds RemoteRegistry on the shared CachedCatalogRegistry pipeline, makes
it cache-first with a TTL, merges its entries over the bundled catalog rather than
replacing them, and finally wires registry_url into ModelManager so the setting
reaches discovery.

Changes

  • src/modeldock/adapters/registry/remote.py — rewritten as a CachedCatalogRegistry
    subclass. Caches to <cache_dir>/remote_catalog_cache.json with a 1-hour TTL, read
    cache-first so a configured URL costs no network round-trip per CLI invocation;
    falls back to an expired cache before giving up, and refresh() bypasses the TTL.
    Remote entries are merged over bundled (remote wins a name collision, no bundled
    model is ever dropped). Adds alias/case-aware lookup, per-entry error tolerance so
    one malformed record cannot discard a whole payload, honest describe() reporting,
    http(s)-only URL validation, and a size-capped streamed response body.
  • src/modeldock/adapters/registry/bundled.py — extracted catalog_entry_to_spec and
    load_bundled_catalog so the entry coercion is shared instead of reached for through
    BundledRegistry._to_spec. BundledRegistry behaviour is unchanged.
  • src/modeldock/core/manager.pycatalog_source="remote" selects RemoteRegistry
    (raising ConfigError when no registry_url is set); under "auto" a configured URL
    merges in ahead of every other source. An unusable URL degrades to a warning rather
    than breaking discovery.
  • src/modeldock/common/config.py"remote" added to the catalog_source allow-list.
  • src/modeldock/adapters/registry/composite.pydescribe() deduplicates sources by
    name. Without this, modeldock sources would list "Bundled" twice under "auto",
    since RemoteRegistry merges bundled internally and the base registry may also be it.
  • tests/unit/test_remote_registry.py — new, 35 tests.
  • Docs: Architecture.md (§9 registry bullet + config paragraph), Development.md,
    CHANGELOG.md ([Unreleased]), and the catalog_source tables in README.md /
    QUICKSTART.md, which enumerated the valid values and were incomplete without
    remote.

Note on one design decision

The CachedCatalogRegistry base is network-first, with the cache only as an offline
fallback. RemoteRegistry deliberately overrides _load/_remote_entries to be
cache-first (fresh cache → network → stale cache → bundled). Network-first would leave
the TTL decorative and keep a round-trip in front of every search/list — which was
one of the original defects. The departure is documented in the method docstring.

Testing

  • pytest647 passed, 4 skipped (skips pre-existing: Ollama CLI not installed).
  • tests/unit/test_remote_registry.py — 35 unit tests driving a real HTTPServer on a
    real socket rather than mocking httpx, with a request counter. The counter is the
    point: a registry that re-fetches per construction passes a mocked test and still
    hits the network on every command. Covers fetch/parse, alias and case resolution,
    search/recommend/by_category reaching remote entries, merge with bundled, remote
    winning a name collision, cache written, second construction served with zero
    further requests, TTL expiry re-fetching, expired-cache fallback when the server is
    down, bundled-only when there is neither, malformed entry skipped, error status and
    unusable payloads degrading, oversized body refused, non-http(s) URLs rejected,
    refresh() bypassing the TTL and surviving failure, describe() accuracy, and the
    ModelManager wiring for both "remote" and "auto".
  • ruff check src tests — All checks passed. ruff format --check — 116 files already
    formatted.
  • mypy --strict src — Success, no issues in 80 source files.
  • bandit -c pyproject.toml -r src — 0 issues (0 high / 0 medium / 0 low).
  • Manual end-to-end through the real CLI against a stub catalog server:
    modeldock search finds the fresh model stamped Remote registry; modeldock info
    resolves it by alias; bundled entries remain present and correctly stamped; a second
    command issues no second HTTP request; modeldock sources reports the remote's
    own contribution (1) separately from bundled (19); modeldock sources refresh
    bypasses the TTL.

Checklist

  • Branch named per Git Workflow (feature/, fix/, docs/, refactor/, test/, chore/)
  • Not developed on main
  • Code follows AGENT.md coding standards (type hints, Pydantic v2, no generic Exception, no business logic in CLI)
  • domain/ and ports/ stay pure (no I/O, no framework imports)
  • Quality gates pass locally: ruff, mypy --strict, bandit, pytest
  • Docs updated if behavior changed
  • pyproject.toml and src/modeldock/__init__.py versions match (if release-related)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Registry: remote refresh with caching

1 participant