Skip to content

The VST cache fill is probed twice and the two answers disagree #11

Description

@stef9github

What happens

/topology shows the vst-local-cache node as unknown while GET /api/storage/vst reports localCacheFillPercent: 44 for the same figure, at the same moment, on the same cluster.

Why it is two answers

The number comes from a df exec against the VST sensor pod, and two independent call sites make that exec:

  • src/lib/pipeline/aggregator.ts — feeds the /topology node
  • src/app/api/storage/vst/route.ts — feeds the storage panel

Only the second one succeeds. The aggregator falls into its fillPct === null branch and renders unknown, with no warning emitted (warnings: []).

Not a permissions problem

Ruled out by A/B: the node reads unknown with cluster-wide pods/exec restored as well as with the narrowed namespaced grant. The other exec-backed probes in the same aggregator pass at the same time — vss-vios-postgres and vss-redis both report ok.

The likely cause is the aggregator's CALL_TIMEOUT_MS, which the route does not share, but that has not been confirmed.

Why it is worth fixing rather than tuning

An operator reading /topology sees a storage cache whose fill is unknown, and an operator reading the storage panel sees 44%. Neither page says the other exists. The underlying issue is that one figure has two probes; the fix is probably one probe with one timeout, not a longer timeout in two places.

Where to start

runInPod in src/lib/k8s.ts, and the cacheFill/cacheHealth derivation around aggregator.ts:855 and :1035. The else branch at :1044 is what renders unknown and is silent — an emitted warning there would have made this visible without a side-by-side comparison.

Found while narrowing RBAC; it predates that change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions