Skip to content

test: preserve add-field insert errors and retry schema mismatches once - #55228

Open
zhuwenxing wants to merge 1 commit into
milvus-io:masterfrom
zhuwenxing:fix-chaos-add-field-retry-1106
Open

zhuwenxing wants to merge 1 commit into
milvus-io:masterfrom
zhuwenxing:fix-chaos-add-field-retry-1106

Conversation

@zhuwenxing

Copy link
Copy Markdown
Contributor

AddFieldChecker discards the insert error after a successful AddCollectionField call and continues to query. Failed inserts can therefore appear as "Unknown error" or be masked by a later query error.

Preserve the exception returned by insert_data. For a typed SchemaMismatchRetryableException, invalidate the schema cache for the client's active database and retry the insert once with regenerated data. Do not repeat the DDL. Return the insert error before querying when the insert still fails; ordinary errors and ambiguous timeouts are not retried.

Validation:

  • 10 temporary focused checks passed, covering bounded retries, active-database cache invalidation, original error retention, and failure paths. These validation helpers are excluded from the PR.
  • On server commit 105fa7a with PyMilvus 3.1.0rc98, the existing warm-schema add-field case and a focused checker SDK check passed. The latter injected one typed client exception before a real insert and verified two insert attempts, one added field, and visible rows; it does not reproduce a server-side schema race.
  • Ruff 0.15.11 check, format --check --diff, configuration smoke check, py_compile, and git diff --check passed. This Python-only change is outside the repository license workflow's internal/**/*.go scope.
  • The concurrently selected original TTL-switch test still failed (expected [1], actual []); its assertion and implementation are unchanged.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
@sre-ci-robot sre-ci-robot added the do-not-merge/work-in-progress Don't merge even CI passed. label Nov 6, 2026
@sre-ci-robot sre-ci-robot added the size/S Denotes a PR that changes 10-29 lines. label Nov 6, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from zhuwenxing after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify Bot added dco-passed DCO check passed. kind/bug Issues or changes related a bug labels Nov 6, 2026
@mergify

mergify Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@zhuwenxing Please associate the related issue to the body of your Pull Request. (eg. "issue: #")

@zhuwenxing
zhuwenxing marked this pull request as ready for review November 6, 2026 07:25
@sre-ci-robot sre-ci-robot removed the do-not-merge/work-in-progress Don't merge even CI passed. label Nov 6, 2026
@zhuwenxing zhuwenxing changed the title fix: preserve add-field insert errors and retry schema mismatches once test: preserve add-field insert errors and retry schema mismatches once Nov 6, 2026
@mergify mergify Bot added the kind/test label Nov 6, 2026
@sre-ci-robot

Copy link
Copy Markdown
Contributor

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

  • /ci-rerun-code-check // for ci-v2/code-check
  • /ci-rerun-code-check-macos // for Code Checker MacOS (GitHub Actions)
  • /ci-rerun-build // for ci-v2/build
  • /ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
  • /ci-rerun-buildenv // for ci-v2/build-env (build milvus-env builder images; update .env after the new tag is ready)
  • /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
  • /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
  • /ci-rerun-ut-cpp // for ci-v2/ut-cpp
  • /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
  • /ci-rerun-e2e-default // for ci-v2/e2e-default
  • /ci-rerun-e2e-pulsar-wp // for ci-v2/e2e-pulsar-wp (GCP branch-equivalent Pulsar + Woodpecker Python E2E)
  • /ci-rerun-e2e-amd // for ci-v2/e2e-amd (e2e pool dispatcher)
  • /ci-rerun-e2e-dist-wp // for ci-v2/e2e-dist-wp (Tencent distributed woodpecker-service boundary)
  • /ci-rerun-build-ut-cov // for ci-v2/build-ut-cov (build + unit tests in one pipeline)
  • /ci-rerun-build-ut-cov-tcus // force TC-US build-ut-cov; respects Portal limit, never falls back to AWS
  • /ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests, ARM)
  • /ci-rerun-gosdk-std // for ci-v2/go-sdk-std (Go SDK E2E, standalone)
  • /ci-rerun-gosdk-dist-wp // for ci-v2/go-sdk-dist-wp (Go SDK E2E, distributed + Woodpecker service)

If you have any questions or requests, please contact @zhikunyao.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test ci-passed dco-passed DCO check passed. kind/bug Issues or changes related a bug kind/test sig/testing size/S Denotes a PR that changes 10-29 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants