Skip to content

test: wait for indexed sources before mix compaction - #55252

Merged
xiaofan-luan merged 1 commit into
milvus-io:masterfrom
xiaofan-luan:codex/fix-compaction-index-readiness-20261106
Nov 7, 2026
Merged

xiaofan-luan merged 1 commit into
milvus-io:masterfrom
xiaofan-luan:codex/fix-compaction-index-readiness-20261106

Conversation

@xiaofan-luan

Copy link
Copy Markdown
Collaborator

The MixCompaction E2E can observe stable Flushed/L1/sorted sources while their vector indexes are still building. Index-based compaction then filters out every source, returns compaction_id=-1, and the test fails its required-plan assertion.

This adds an opt-in index-readiness check to the existing source waiter. The named index must be Finished, have no pending rows, and cover the current source snapshot before two stable observations count. The index RPC shares the existing deadline; failed indexes fail immediately with their reason. Other callers retain the existing behavior.

The MixCompaction test now uses an explicit schema and named FLAT index, verifies the actual index type, and enables the stronger wait. Existing plan, topology, row-count and primary-key assertions remain intact. Server-independent regressions are included in the existing compaction-integrity unit entrypoint.

Observed failure: #54582, GCP distributed Woodpecker E2E #72022. Service logs showed all three source indexes still InProgress when manual compaction returned zero tasks.

Validation

  • 13 focused helper regressions passed. The regressions also fail against the old helper when it is adapted to accept and ignore the new keyword, confirming a behavioral failure rather than a signature mismatch.
  • Ruff lint and format checks passed for all changed Python files.
  • bash -n tests/scripts/ci_compaction_integrity_unit.sh and git diff --check passed.
  • Full cluster E2E was not run locally: no Milvus service is available and the local full-suite Python/SDK versions differ from CI. A dedicated Pulsar/Woodpecker E2E run will be requested for this PR.

Focused local command:

PYTHONPATH=tests/python_client python3 -m pytest \
  --noconftest -c /dev/null -p no:cacheprovider \
  -o 'markers=tags: case tags' -q \
  tests/python_client/common/compaction_index_readiness_helper_tests.py

Signed-off-by: xiaofanluan <xf@hjjaq.com>

Copy link
Copy Markdown
Collaborator Author

/ci-rerun-e2e-pulsar-wp

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T01:29:50.155897Z 394e5d7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sre-ci-robot sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. area/test sig/testing labels Nov 7, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from xiaofan-luan after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify Bot added dco-passed DCO check passed. kind/test labels Nov 7, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

  • /ci-rerun-code-check // for ci-v2/code-check
  • /ci-rerun-code-check-macos // for Code Checker MacOS (GitHub Actions)
  • /ci-rerun-build // for ci-v2/build
  • /ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
  • /ci-rerun-buildenv // for ci-v2/build-env (build milvus-env builder images; update .env after the new tag is ready)
  • /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
  • /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
  • /ci-rerun-ut-cpp // for ci-v2/ut-cpp
  • /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
  • /ci-rerun-e2e-default // for ci-v2/e2e-default
  • /ci-rerun-e2e-pulsar-wp // for ci-v2/e2e-pulsar-wp (GCP branch-equivalent Pulsar + Woodpecker Python E2E)
  • /ci-rerun-e2e-amd // for ci-v2/e2e-amd (e2e pool dispatcher)
  • /ci-rerun-e2e-dist-wp // for ci-v2/e2e-dist-wp (Tencent distributed woodpecker-service boundary)
  • /ci-rerun-build-ut-cov // for ci-v2/build-ut-cov (build + unit tests in one pipeline)
  • /ci-rerun-build-ut-cov-tcus // force TC-US build-ut-cov; respects Portal limit, never falls back to AWS
  • /ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests, ARM)
  • /ci-rerun-gosdk-std // for ci-v2/go-sdk-std (Go SDK E2E, standalone)
  • /ci-rerun-gosdk-dist-wp // for ci-v2/go-sdk-dist-wp (Go SDK E2E, distributed + Woodpecker service)

If you have any questions or requests, please contact @zhikunyao.

@xiaofan-luan
xiaofan-luan merged commit aec29e9 into milvus-io:master Nov 7, 2026
3 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test dco-passed DCO check passed. kind/test sig/testing size/L Denotes a PR that changes 100-499 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants