DAOS-17535 chk: misc improvements for CR logic #17427

Nasf-Fan · 2026-01-23T08:07:51Z

Include the followings:

When create CHK IV namespace, make the secondary group to be same as the primary group. Otherwise, CHK logic may hit DER_NONEXIST trouble when communicate via IV.
Integrate CHK IV namespace create and destroy API, cleanup related logic, redefine the version.
Get ranks list and IV namespace version from CHK leader when rejoin. Adjust CHK_REJOIN RPC for related changes.
Remove unsupported functionality for checking the specified 'phase'.
Add new test for case of lost some engine(s) before start checker.
Dedicated ULT to handle dead rank event, that will not be affected by checker start or stop. Then even if check scheduler exited, the subsequent check query still can work against the latest rank list.

Test-tag: recovery

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-01-23T08:08:10Z

Ticket title is 'DAOS checker cannot completed on Aurora after some engines excluded'
Status is 'In Review'
Labels: 'scrubbed_2.6.5'
https://daosio.atlassian.net/browse/DAOS-17535

daosbuild3 · 2026-01-23T21:46:19Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/1/execution/node/1302/log

daosbuild3 · 2026-01-23T22:11:11Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/1/execution/node/1317/log

daosbuild3 · 2026-01-24T05:22:33Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17427/3/testReport/

daosbuild3 · 2026-01-24T22:31:13Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/3/execution/node/1283/log

daosbuild3 · 2026-01-25T03:49:07Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17427/5/display/redirect

daosbuild3 · 2026-01-25T10:29:00Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/7/execution/node/1318/log

daosbuild3 · 2026-01-25T11:37:10Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/7/execution/node/1308/log

daosbuild3 · 2026-01-25T18:44:09Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17427/9/testReport/

daosbuild3 · 2026-01-25T23:08:34Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/9/execution/node/1324/log

daosbuild3 · 2026-01-25T23:21:13Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17427/9/execution/node/1314/log

Include the followings: 1. When create CHK IV namespace, make the secondary group to be same as the primary group. Otherwise, CHK logic may hit DER_NONEXIST trouble when communicate via IV. 2. Integrate CHK IV namespace create and destroy API, cleanup related logic, redefine the version. 3. Get ranks list and IV namespace version from CHK leader when rejoin. Adjust CHK_REJOIN RPC for related changes. 4. Remove unsupported functionality for checking the specified 'phase'. 5. Add new test for case of lost some engine(s) before start checker. 6. Dedicated ULT to handle dead rank event, that will not be affected by checker start or stop. Then even if check scheduler exited, the subsequent check query still can work against the latest rank list. Test-tag: recovery Signed-off-by: Fan Yong <[email protected]>

daltonbohning · 2026-01-26T16:32:56Z

src/tests/ftest/recovery/cat_recov_core.yaml

+pool:
+  scm_size: 6G
+  nvme_size: 80G


Do we need to specify these individually or could we just specify size: and let DAOS use the default ratio?

I tried with size: 180G, but failed to create the pool during CI test because of size trouble.

I think that the issue is from test_setup_pool_create() that is used for most of tests under suite. Similar configuration in src/tests/ftest/daos_test/suite.yaml.

Nasf-Fan · 2026-01-27T15:59:27Z

Ping reviewers, this patch is important for 2.8 release, please help to view, thanks!

Nasf-Fan · 2026-01-27T16:00:37Z

All required CR tests have passed.

gnailzenh · 2026-01-28T02:53:46Z

src/gurt/misc.c

+ */
+
+bool
+d_rank_list_bsearch(d_rank_list_t *rank_list, d_rank_t rank, int *idx)


why don't you just replace the d_rank_list_find() with this implementation?

d_rank_list_find is used for finding rank in any rank list in spite of sorted or not. If I replace it but the caller does not sort related rank list before calling it, then the result will be wrong. So I create new API d_rank_list_bsearch() that is only used for the sorted rank list.

gnailzenh · 2026-01-28T02:59:39Z

src/chk/chk_internal.h

+	    ci_starting : 1, ci_stopping : 1, ci_started : 1, ci_inited : 1, ci_pause : 1,
+	    ci_skip_oog : 1, ci_rejoining : 1, ci_implicated : 1;
+	uint32_t ci_start_flags;
+	uint32_t ci_ns_ver;


I really don't like the auto-change like this...

Me too, but clang-format require that by force. That is one of the reason for the huge patch.

gnailzenh · 2026-01-28T03:02:33Z

src/chk/chk_engine.c

+	}

-	D_ASSERT(rank_list != NULL);
+	if (ins->ci_ranks == NULL) {


is there other non-leader case that we can see ci_ranks == NULL?

Before the first time starting CR, the list will be empty.

gnailzenh · 2026-01-28T03:06:10Z

src/chk/chk_engine.c

+		}
+	}
+
+	if (unlikely(ins->ci_ranks == NULL))


so we don't need the "if" anymore, since we just created it, or chk_prop_fetch can still return NULL in non-failure case?

If CR has never run before, then rank list will be empty.

src/chk/chk_internal.h

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7_t branch 2 times, most recently from 562010a to ed02f46 Compare January 24, 2026 03:57

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7_t branch 3 times, most recently from efa47b4 to a944ece Compare January 25, 2026 03:47

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7_t branch from a944ece to 3e235f3 Compare January 25, 2026 05:12

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7_t branch 2 times, most recently from e741d4e to f684f3f Compare January 25, 2026 17:14

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7_t branch from f684f3f to ae6132f Compare January 26, 2026 05:45

Nasf-Fan marked this pull request as ready for review January 26, 2026 13:58

Nasf-Fan requested review from a team as code owners January 26, 2026 13:58

Nasf-Fan requested review from daltonbohning, gnailzenh and wangshilong January 26, 2026 14:02

Nasf-Fan requested a review from kjacque January 26, 2026 14:02

Nasf-Fan mentioned this pull request Jan 26, 2026

DAOS-17535 chk: misc improvements for CR logic #17329

Closed

6 tasks

daltonbohning reviewed Jan 26, 2026

View reviewed changes

Nasf-Fan mentioned this pull request Jan 27, 2026

DAOS-18238 chk: handle CRT_EVS_GRPMOD event from CaRT PG #17459

Open

6 tasks

Nasf-Fan added the priority Ticket has high priority (automatically managed) label Jan 27, 2026

wangshilong approved these changes Jan 28, 2026

View reviewed changes

gnailzenh approved these changes Jan 28, 2026

View reviewed changes

gnailzenh merged commit 4c8fb91 into master Jan 28, 2026
40 checks passed

gnailzenh deleted the Nasf-Fan/DAOS-17535_7_t branch January 28, 2026 03:52

DAOS-17535 chk: misc improvements for CR logic #17427

DAOS-17535 chk: misc improvements for CR logic #17427

Uh oh!

Conversation

Nasf-Fan commented Jan 23, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Jan 23, 2026

Uh oh!

daosbuild3 commented Jan 23, 2026

Uh oh!

daosbuild3 commented Jan 23, 2026

Uh oh!

daosbuild3 commented Jan 24, 2026

Uh oh!

daosbuild3 commented Jan 24, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

daosbuild3 commented Jan 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jan 27, 2026

Uh oh!

Nasf-Fan commented Jan 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants