Skip to content

fix(wpcli): improve segment inspection and scan diagnostics - #419

Merged
tinswzy merged 1 commit into
zilliztech:masterfrom
tinswzy:fix/wp-cli-segment-diagnostics-418
Oct 14, 2026
Merged

tinswzy merged 1 commit into
zilliztech:masterfrom
tinswzy:fix/wp-cli-segment-diagnostics-418

Conversation

@tinswzy

@tinswzy tinswzy commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

When an Active segment has no loaded writer or discoverable local directory, scan/inspect/probe cannot select its storage instance from coordinator metadata alone. Scan hides the per-replica errors behind an unknown summary, while a local-only successful survey can overstate end-to-end readability.

This change adds paired --bucket / --root-path flags to all three commands and forwards the escaped instance location to each quorum member. Omitting both retains existing node-side discovery.

Scan now exposes per-replica local ranges, survey stop reasons, and HTTP/transport/response errors in text and structured output. All-replica confirmed local emptiness on an Active segment is distinguished from incomplete or unavailable inspection results. A missing range remains unknown while any replica is unavailable or has an incomplete survey: a short observed copy cannot rule out a healthy uninspected tail. Raw checksum failures are not labelled as local emptiness, and responses lacking a survey stop reason are rejected as unusable.

Probe no longer declares a permanent read stall solely from all replicas reporting no local data for an Active segment. Finalized-segment shortfall checks remain in place. Help, summaries and the CLI README clarify that scan surveys local coverage/blocks; it does not verify pending writes, shared compacted objects, or application progress.

Validation:

  • go build -o /tmp/woodpecker-wp-418 ./cmd/wpcli
  • go test -race ./cmd/wpcli/... -count=1 -timeout=3m
  • golangci-lint run --config .golangci.yml --timeout=3m ./cmd/wpcli/...
  • Regression tests cover paired flag validation and escaped forwarding, confirmed empty Active vs finalized absence, checksum-failed blocks, unusable responses, preserved replica error details and JSON per-replica coverage.

Based directly on master; independent of #417.

Closes #418

@github-actions

Copy link
Copy Markdown

CI commands

Comment any of these to rerun checks on this pull request:

Command Reruns
/rerun every failed workflow on the latest commit
/rerun-lint Lint
/rerun-mod-tidy Go Mod Tidy Check
/rerun-unit-test Unit Test
/rerun-e2e-local E2E Local
/rerun-e2e-objectstorage E2E Object Storage
/rerun-e2e-service E2E Service
/rerun-component-test Component Integration Test
/rerun-chaos Chaos Test
/rerun-service-stability-process Service Stability (Process)
/rerun-service-stability-docker Service Stability (Docker)
/rerun-operator Operator Test
/rerun-upgrade-local Upgrade Compat Local
/rerun-upgrade-objectstorage Upgrade Compat Object Storage
/rerun-upgrade-service Upgrade Compat Service

The ci-passed label is applied automatically once every required check passes.

@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.84536% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.12%. Comparing base (c45123a) to head (3089e71).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
cmd/wpcli/cmd/log_scan.go 90.00% 2 Missing and 1 partial ⚠️
cmd/wpcli/cmd/segment_inspect.go 88.88% 1 Missing ⚠️
cmd/wpcli/cmd/segment_probe.go 93.75% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #419      +/-   ##
==========================================
+ Coverage   84.06%   84.12%   +0.06%     
==========================================
  Files         214      214              
  Lines       30214    30293      +79     
==========================================
+ Hits        25398    25483      +85     
+ Misses       3640     3637       -3     
+ Partials     1176     1173       -3     
Components Coverage Δ
Server 83.69% <ø> (+<0.01%) ⬆️
Client 90.86% <ø> (+0.06%) ⬆️
Meta 85.58% <ø> (ø)
Common 88.23% <ø> (-0.07%) ⬇️
Files with missing lines Coverage Δ
cmd/wpcli/cmd/log_scan_reconcile.go 99.15% <100.00%> (+2.08%) ⬆️
cmd/wpcli/cmd/segment_inspect.go 83.78% <88.88%> (+0.65%) ⬆️
cmd/wpcli/cmd/segment_probe.go 88.03% <93.75%> (+0.85%) ⬆️
cmd/wpcli/cmd/log_scan.go 78.60% <90.00%> (+2.47%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@czs007

czs007 commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Must-fix issues introduced by this PR: 0
Merge recommendation: Mergeable as-is — this review found no must-fix issue introduced by this PR.

One-sentence overview, then one item under Medium, then the pre-existing section.

Adversarial review of the --bucket/--root-path location plumbing found one path that the new flags do not reach: the orphan-segment gap probe still issues its inspect request without a location, which can turn a held orphan into a false "data is gone" conclusion.

Medium

  • cmd/wpcli/cmd/log_scan.go:111 — The gap-probe closure passed to reconcileScanWithGaps calls probeGapForData without forwarding the location variadic that runLogScan now accepts, so probeGapForData (line 175) builds its /admin/logstore/segment/inspect request (line 182) without bucket_name/root_path. The other three inspect/probe paths (scanSegmentNodes, inspectEachNode, probeEachNode) do forward it, and the README now documents that log scan accepts these flags to select an instance. Failure: an orphan segment id (missing from metadata) has its data directory on a node where several storage instances hold the same log_id/segment_id directory, or local auto-discovery fails, and the operator runs wp log scan <log> --bucket X --root-path Y → the server's resolveProbeInstance returns "held by N instances; pass bucket_name and root_path" as HTTP 404, probeGapForData treats non-200 as "this node has nothing" (asked=true, continue, lines 188-190), and the scan reports "segment id N is missing from metadata and no node holds data for it … the record is gone rather than the data" (log_scan_reconcile.go:410-413) for data that a node actually holds. Suggestion: add a location ...scanLocation parameter to probeGapForData, append bucket_name/root_path the same way the other three paths do, forward location... from the closure, and add a gap-probe location-forwarding assertion to segment_location_test.go (which currently covers only the regular scan/inspect/probe paths). (raised by sijie-ni-0214, xiaocai2333)

Pre-existing hazards — not introduced by this PR (follow-up only)

The following hazards are real but pre-existing and were not introduced by this PR. They should be tracked as separate follow-up work and do not block this PR.

  • cmd/wpcli/cmd/log_scan_reconcile.go:251 — The "behind / need resyncing" block subtracts each answered replica's coverage from the union held (built at line 210) without checking node.completeClaim(), even though scanNode.stopped (lines 49-51) and completeClaim (lines 64-66) define coverage as a claim only when the survey reached the end of the segment, and a lower bound otherwise. Failure: a Completed/Active segment with two replicas where node-1's inspect stops early on max_blocks (coverage 0-99) and node-2 reaches the end (coverage 0-199) → the scan prints "node-1 is missing 100-199, which another replica holds — those replicas are behind and need resyncing", reporting a healthy but partially surveyed replica as behind and potentially prompting an unnecessary resync. A fix would restrict the behind judgement to replicas where completeClaim() is true, or report "survey bounded" when !settled. Unrelated to this PR because the diff touches scanNode fields, confirmedLocalEmpty, the settled initialisation and replica loop, the open-segment and !settled branches, and boundedReasons, while the behind block at lines 243-260 and the construction of held are unchanged line for line, so behaviour is identical before and after this PR. No tracking issue yet. (raised by sijie-ni-0214, xiaocai2333)

@tinswzy
tinswzy merged commit 9963a5d into zilliztech:master Oct 14, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enhance(wpcli): Improve segment inspection and log scan diagnostics

2 participants