DAOS-19386 dfuse: move notify calls to an asynchronous queue - #18806
Draft
mjmac wants to merge 3 commits into
Draft
DAOS-19386 dfuse: move notify calls to an asynchronous queue#18806mjmac wants to merge 3 commits into
mjmac wants to merge 3 commits into
Conversation
Don't allow a stuck child process to tank the entire NLT
run. When a stall is detected, attempt to capture stack
traces and fail early.
The /proc dump shows a real stall directly. From the hang that
motivated this patch: the stalled child holds the parent-dir lock
while waiting on its LOOKUP reply, and every dfuse worker is blocked
on that same lock trying to write a dentry invalidation:
TID 34163: comm=daos wchan=request_wait_answer state=S
[<0>] request_wait_answer+0xfa/0x210 [fuse]
[<0>] fuse_simple_request+0x1b8/0x330 [fuse]
[<0>] fuse_lookup_name+0xa4/0x1c0 [fuse]
[<0>] fuse_lookup+0x66/0x190 [fuse]
[<0>] __lookup_hash+0x70/0xa0
[<0>] __filename_create+0x87/0x150
[<0>] do_mkdirat+0x4c/0x160
[<0>] __x64_sys_mkdir+0x47/0x70
TID 29212: comm=dfuse worker wchan=fuse_reverse_inval_entry state=D
[<0>] fuse_reverse_inval_entry+0x40/0x210 [fuse]
[<0>] fuse_notify+0x287/0x500 [fuse]
[<0>] fuse_dev_do_write+0x305/0x4e0 [fuse]
[<0>] fuse_dev_write+0x50/0x80 [fuse]
TID 29213, 29214: identical (the whole worker pool)
Signed-off-by: Michael MacDonald <github@macdonald.cx>
Collaborator
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18806/1/testReport/ |
Most fuse_lowlevel_notify_* calls were already effectively fire-and-forget, but blocked the calling worker while the kernel processed them. In certain scenarios, e.g. mkdir racing with multiple setxattrs in the same directory, this pattern could result in all threads being blocked, and then the mount would hang. Making the notify calls properly asynchronous means that workers will not block on waiting for a response. A dedicated thread now handles queued notifications, with some logic to coalesce notifications. duns_create_path, which does need the invalidation's effect before returning, now polls until the path is confirmed bound to the new container before returning success. Signed-off-by: Michael MacDonald <github@macdonald.cx>
mjmac
force-pushed
the
mjmac/dfuse-notify-v3
branch
from
August 6, 2026 21:12
e77e690 to
3b53090
Compare
Collaborator
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18806/1/execution/node/1267/log |
mjmac
changed the title
Collaborator
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18806/2/testReport/ |
Client-side WARN wasn't enough - the engine's own RPC chatter under churn is the actual byte source, and CI runs it at DEBUG. Runtime `dmg server set-logmasks` lowers it for the test's duration without a restart. NLT never sets a config log_mask, so a bare reset can't restore it; set DEBUG back explicitly instead, matching NLT's startup. Signed-off-by: Michael MacDonald <github@macdonald.cx>
|
Errors are Unable to load ticket data |
Collaborator
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18806/3/testReport/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most fuse_lowlevel_notify_* calls were already effectively
fire-and-forget, but blocked the calling worker while the kernel
processed them. In certain scenarios, e.g. mkdir racing with
multiple setxattrs in the same directory, this pattern could
result in all threads being blocked, and then the mount would hang.
Making the notify calls properly asynchronous means that workers
will not block on waiting for a response. A dedicated thread now
handles queued notifications, with some logic to coalesce notifications.
duns_create_path, which does need the invalidation's effect before
returning, now polls until the path is confirmed bound to the
new container before returning success.
Signed-off-by: Michael MacDonald github@macdonald.cx