[Philip Lombardi (plombardi89)]

Why

CI ran go test without -v, so a skipped test never appeared in its log. A test that skips on every run, because a variable is never set or a tool is never installed, was invisible: it passed review and CI indefinitely. A scan found several, listed below.

What changes

Tests run verbosely, and skips are listed

• make test now runs through gotestsum v1.13.0, pinned as a go.mod tool like govulncheck (go tool gotestsum).
• Its output matches go test -v, and it ends with every skipped and failed test and its reason:

  === Skipped
  === SKIP: deploy/machine-ops TestEnvtestMachineAllocationRefPolicy (0.00s)
      machineallocation_envtest_test.go:30: set KUBEBUILDER_ASSETS for real API server tests
  ...
  DONE 15955 tests, 46 skipped in 168.391s
  

• GOTEST also passes -v, so the other test targets are verbose too.
• gotestsum exits with go test's status, so the change can't hide a failure.

Skipped tests in the CI job summary

• gotestsum also writes the run as JSON to tmp/go-test.json.
• A new tool, hack/cmd/test-summary, turns that into the Test job's summary: the counts, a table of skipped tests with their reasons, and the failures, including packages that failed outside any test (a build error or a TestMain).
• It reports and never judges: it exits non-zero only when the report is unreadable or holds no results.
• The CI step runs even when tests fail, as long as a report exists.

New dependencies: go.mod gains gotestsum and three small modules it needs (gotestdox, dnephin/pflag, google/shlex), all // indirect. No existing version moves, and go mod tidy is stable. NOTICE collects only direct requirements, so it doesn't change.

Tests that should run or fail

• TestWriteInventoryAsMachina: skipped with no reason since it was added. It's a pure function test and it passes, so it runs again.
• TestLinkManager_Integration: deleted. Its body was only the two skips.
• TestResourceMergeRejectsMismatchedSchema: exists to flag an OpenTelemetry SDK change, but skipped when that change happened. It now fails instead.
• TestOverridesValidateShippedExample: skipped if the shipped example was missing. The file is tracked, so a missing file means it moved, and the test now fails.

What the list shows today

From a full local run (46 skips). CI skips 41: it has the frontend build, and its host has no leftover node artifacts.

• Envtest tests that never run in make test:
  • TestEnvtestMachineAllocation{RefPolicy,BootstrapTokenPolicy} (deploy/machine-ops) and TestEnvtest{Allocation,Capacity}MinimalMain (pkg/machineops/controller) run in no CI job.
  • The racer envtest tests also skip here, but the racer-envtest job runs them.
• TestMediaRealToolsOptional (metalman ISO packaging): no CI job installs its tools.
• Opt-in by design: the racer-loadgen diagnostic and performance tests, the overlaybd ACR fixture, and the two racersdk Rust fixtures.
• Root-only: two tests.
• The isolation conformance suite skips the cases a backend doesn't support, which is expected.

Making the envtest and ISO tests run in CI is not in this PR.

Checked and left alone

The scan also covered the 28 places where a test does if err != nil { return }. None hides a failure:

• 25 are fake servers or accept and read loops, which return when the listener closes or the client hangs up.
• One is a load worker that reports its error on a results channel before returning.
• One is a fuzz target discarding input it can't parse.

Testing

• CI=1 make test passes through the new path: 15,955 tests, 46 skipped. Running test-summary on that report gives the same counts.
• In this PR's CI run, the Test job ran the same 15,955 tests with 41 skipped, and the new "Summarize tests" step succeeded.
• hack/cmd/test-summary tests use a real go test -json skip as captured. They cover:
  • counting and sorting;
  • output from parallel tests arriving interleaved;
  • framing lines from older toolchains, which lack OutputType;
  • output not leaking between repeated runs;
  • pipes and long reasons in table cells;
  • failed tests vs. packages that failed outside a test;
  • empty, malformed and missing reports.
• make fmt and make lint are clean. go mod tidy makes no change.

Notes

• The Test job's log grows to about 51k lines. The summary at the end of the log, and the job summary, are where to look.
• Local make test output is now verbose too, as asked. The skip and failure summary is the last thing printed.