Skip to content

Add Gantry test dashboard with live fleet telemetry and disk reclamation - #1113

Open
Vaibhav Patel (vpatelsj) wants to merge 6 commits into
mainfrom
vpatelsj/gantry-dashboard
Open

Vaibhav Patel (vpatelsj) wants to merge 6 commits into
mainfrom
vpatelsj/gantry-dashboard

Conversation

@vpatelsj

Copy link
Copy Markdown
Contributor

Why

Testing Gantry across a large Flex fleet needs more than checking whether Pods are running. We need to see how images move between workers, how quickly the fleet becomes ready, where disk space is going, and whether a cleanup actually reclaimed it.

This adds a standalone dashboard for that workflow, along with the telemetry and image-GC fixes found while testing it on a 1,000-worker AKS Flex cluster.

What changes

A single view of the fleet

  • Show image distribution, Pod lifecycle, fleet readiness, and disk usage together, with independently status-sorted worker grids and synchronized worker selection.
  • Report origin/peer receive rates and cumulative observed bytes. Keep full observed run history across browser refreshes, support zoom/reset, and freeze charts when all workers are Ready or Completed.
  • Keep useful last-known telemetry during agent outages without treating missing observations as zero progress.
  • Make traffic scope explicit: these are cluster-wide receiver counters observed during the selected run, not exact image-specific accounting. History before collection starts is not reconstructed.

Generate and clean up demo runs

  • Generate bounded random-payload images using durable Workload Identity Jobs, then start digest-pinned demo DaemonSets.
  • Offer opt-in ACR streaming conversion. Verify the converted artifact before deployment; regular OCI generation remains the default.
  • Add a confirmed, cluster-wide reclaim action that deletes dashboard-managed demos and prunes unused worker images. Preserve unrelated workloads, pinned images, all container references, and all runtime leases.
  • Wait for deleted demos' stopped runtime container records to disappear before reporting their images reclaimed. Surface retained images and failures rather than reporting false success.
  • Allow dismissal of cleanup progress and automatically retry while a previous coordinator is being removed.

Fix image filesystem reporting

Export the proxy snapshotter's actual storage root to containerd and keep it consistent across the configurator, shell entrypoint, and Helm settings. Without this, kubelet image GC queried a nonexistent plugin directory. The configuration change requires a containerd restart; it does not require replacing live snapshotter Pods.

Keep installation on demand

  • Add a dedicated Containerfile, standalone Helm chart, and Docker/Podman build-push-deploy script targeting GHCR.
  • Require an explicit kubeconfig/context, deploy immutable image digests, and support existing pull Secrets for dashboard/helper images.
  • Document Azure identity, registry permissions, and migration prerequisites. This is separate from Unbounded CI and does not upgrade Gantry or automatically adopt an existing manual installation.

Supporting changes include exact-workload operator overrides for test image rollout, Flex readiness handling, and documented deferred streaming cold-start follow-ups.

Validation

  • Scoped Go formatting and lint passed.
  • Go race tests passed for the dashboard, helper behavior, and Helm render tests.
  • 26 frontend unit tests passed, with desktop/mobile browser regression checks covering the four-panel layout, history, selection, generation, and reclamation.
  • Six deployment-script contract tests passed for Docker/Podman commands, digest pinning, explicit context, and failure handling.
  • Live testing on the 1,000-worker fleet verified telemetry, the image-GC root fix, cleanup execution, and the dashboard layout. A real ACR conversion smoke test verified the streaming artifact before a simulated deployment.

Limitations

Chart history is held in the dashboard process and is lost on Pod restart. Runtime lease reclamation is intentionally out of scope. The new Docker/Podman-to-GHCR publishing path has render/contract coverage but has not been exercised end to end because neither engine was installed on the development machine. Azure federation, GHCR login, and any migration from the manual deployment remain explicit setup steps.

Vaibhav Patel and others added 5 commits November 10, 2026 11:37
Add the standalone Gantry UI, incremental worker updates, cluster transfer activity, startup timing, filesystem usage, and on-demand random image builds. Preserve last successful observations during agent failures.\n\nAdd exact workload-name targeting for operator overrides, improve Flex component readiness handling, and document deferred streaming cold-start follow-ups.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Export the host-visible snapshotter data root through containerd proxy-plugin metadata. Wire custom roots through the configurator and Helm chart, preserve managed-config migration and restore behavior, and cover filesystem reporting with regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add durable fleet reclamation with protected runtime references, delayed container cleanup retries, dismissible progress, and automatic coordinator replacement retries. Add per-image ACR streaming conversion with verified artifact deployment and regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… charts

Show distribution, lifecycle, readiness, and storage together with status-sorted 25-column grids. Add observed run traffic totals and history, preserve collection across browser reconnects, and freeze charts when all workers are ready or completed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a standalone Helm chart, Docker/Podman build-push-deploy workflow, and dedicated container image. Support digest-pinned GHCR helper images and existing pull secrets while keeping generated image bases in ACR. Document identity, authentication, deployment, and migration prerequisites with render and workflow tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@vpatelsj
Vaibhav Patel (vpatelsj) requested a review from a team November 11, 2026 04:37
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant