DAOS-16759 test: extend_simple wait for rebuild start before FI clear#18662
DAOS-16759 test: extend_simple wait for rebuild start before FI clear#18662kccain wants to merge 1 commit into
Conversation
Adjust the daos_extend_simple kill path (e.g., for EXTEND14 test) to wait for the combined extend+exclude rebuild to start before clearing the rebuild scan hang fault injection. And add debug print messages around key events in the test sequence. Test-tag: test_daos_extend_simple Skip-unit-tests: true Skip-fault-injection-test: true Skip-test-rpms: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
|
Ticket title is 'daos_test/suite.py:DaosCoreTest.test_daos_extend_simple - rebuild test timeout' |
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18662/1/execution/node/900/log |
|
Ready for review after build 1 passed the daos_test -B (daos_extend_simple.c) tests https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18662/1/artifact/Functional%20Hardware%20Medium%20Verbs%20Provider%20MD%20on%20SSD/daos_test/suite.py/results.json |
| test_arg->pool.pool_info.pi_rebuild_st.rs_version, pre_op, cb_arg->rank, opc, | ||
| extend_opstrs[opc]); | ||
|
|
||
| sleep(10); |
There was a problem hiding this comment.
[minor] this sleep could be removed now?
There was a problem hiding this comment.
The sleep is intentional. The wait above uses rs_version advancement as a signal that a new rebuild has started, but it doesn't guarantee the engine-side task has moved from scheduled in the queue to executing and reaching the scan hang path. So the sleep keeps the test from racing ahead too fast, in an effort to make the rest of the sequencing including fault-clear more deterministic.
|
|
||
| print_message("extend_cb_internal: clear FI via daos_debug_set_params()\n"); | ||
| rc = daos_debug_set_params(test_arg->group, -1, DMG_KEY_FAIL_LOC, 0, 0, NULL); | ||
| assert_success(rc); |
There was a problem hiding this comment.
[Question] Would you explain a bit why this and above change could avoid Rebuild wait test timeout? changes makes sense to me, but I did not get why it might hang rebuild before.
There was a problem hiding this comment.
The intent of the new wait is to confirm that the second rebuild phase (combined extend + exclude) has started after killing rank 2, so we know map/update processing has progressed before issuing the all-ranks fault-clear call. I have seen daos_debug_set_params() delays in other kill-then-clear test patterns, so restructuring this sequence is still valuable to reduce that risk.
With this change in place, a 25x repeat run in the debug-only PR still showed two high timing outliers (repeat004 and repeat023), although both passed: https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18642/3/artifact/Functional%20Hardware%20Medium%20Verbs%20Provider%20MD%20on%20SSD/daos_test/suite.py/
In those outliers, the 2-3 minute delay occurred mostly before reaching the fault-clear call, during extend_write(). Client object RPC activity (including URI/HG lookup paths) showed dead-rank related retry behavior, with timeout or OOG outcomes depending on contact-rank selection. After membership converged, the test completed successfully. So for these specific outliers, the dominant delay appears to be pre-clear extend/rebuild progression, not the fault-clear call itself.
I am continuing to investigate this with @frostedcmos in the same debug-only PR, where we are combining this test sequencing change with URI/HG lookup debug changes: #18642
There was a problem hiding this comment.
Thanks for explaining. So it sounds like waiting rebuild here makes sure rank system membership has been changed to excluded, and daos_debug_set_params() will skip to retry again and again for exclude ranks, It sounds like we are trying to walkaround the issues somehow. In theory daos_debug_set_params() should see system memship change with its retry and succeed finally(time should be similar just like we wait rebuild started and retry again.).
3 participants
Adjust the daos_extend_simple kill path (e.g., for EXTEND14 test) to wait for the combined extend+exclude rebuild to start before clearing the rebuild scan hang fault injection. And add debug print messages around key events in the test sequence.
Test-tag: test_daos_extend_simple
Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-test-rpms: true
Steps for the author:
After all prior steps are complete: