NAS-142112 / None / Fix nfsd client teardown hang from leaked callback count - #333
Merged
Conversation
The nfsd4_callback workqueue jobs exist to queue backchannel RPCs to rpciod. Because they run in different workqueue contexts, the rpc_task can run concurrently with the workqueue job itself, should it become requeued. This is problematic as there is no locking when accessing the fields in the nfsd4_callback. Add a new unsigned long to nfsd4_callback and declare a new NFSD4_CALLBACK_RUNNING flag to be set in it. When attempting to run a workqueue job, do a test_and_set_bit() on that flag first, and don't queue the workqueue job if it returns true. Clear NFSD4_CALLBACK_RUNNING in nfsd41_destroy_cb(). This also gives us a more reliable mechanism for handling queueing failures in codepaths where we have to take references under spinlocks. We can now do the test_and_set_bit on NFSD4_CALLBACK_RUNNING first, and only take references to the objects if that returns false. Most of the nfsd4_run_cb() callers are converted to use this new flag or the nfsd4_try_run_cb() wrapper. The main exception is the callback channel probe, which has its own synchronization. Signed-off-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> (cherry picked from commit 1054e8f)
nfsd_break_one_deleg() sets NFSD4_CALLBACK_RUNNING via test_and_set_bit at entry to serialize recall work, then calls nfsd4_run_cb() to queue the recall. When the queue attempt fails the refcount bump is undone, but the RUNNING bit is left set. The only site that clears the bit is nfsd41_destroy_cb() (fs/nfsd/nfs4callback.c), which runs from the workqueue and is therefore unreachable when nothing was queued. The bit becomes a permanent latch on dp->dl_recall.cb_flags: every subsequent break_lease() on the same delegation hits the early-return guard in nfsd_break_one_deleg() and silently skips the recall, so the delegation is never broken and the conflicting open or lock stalls. Fix by clearing NFSD4_CALLBACK_RUNNING on the !queued branch alongside the refcount_dec. Fixes: 1054e8f ("nfsd: prevent callback tasks running concurrently") Cc: stable@vger.kernel.org Signed-off-by: Jeff Layton <jlayton@kernel.org> Link: https://patch.msgid.link/20260526-cb_recall_any_callback_running_stuck-v1-2-310011a028f3@kernel.org Signed-off-by: Chuck Lever <chuck.lever@oracle.com> (cherry picked from commit fa183d59e531e016d4495e21f00d80c63aebd907)
bugclerk
changed the title
yocalebo
approved these changes
Aug 7, 2026
|
Not updating JIRA ticket https://ixsystems.atlassian.net/browse/NAS-142112 target versions as no JIRA version corresponds to this PR |
|
This PR has been merged and conversations have been locked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Support raised SEE-589 after a customer's NFS server (~87 NFSv4 clients) wedged for the third time. Once it happens nfsd can't expire clients any more, every later client teardown piles up behind the stuck one, and since the stuck task is the laundromat (nfsd's periodic worker that expires dead clients and cleans up NFSv4 state), all of that cleanup stops too.
rpc.nfsdwon't die on SIGKILL at that point, so only a reboot clears it.This isn't new in 25.10.4. As per the attached debugs it hit twice on 6.12.15 (Mar 4 and Jun 20) and once on 6.12.91 (Jul 9), with the same stack every time:
Root cause
NFSv4 hands clients delegations so they can cache a file locally, and the server has to call the client back to take one away. Each of those callbacks takes a count on
cl_cb_inflight, and onlynfsd41_destroy_cb()drops it again.There is one callback object per delegation, so if a delegation gets broken twice, both breaks queue the same work item. The first callback RPC fails,
nfsd4_cb_release()tries to requeue it for a retry,nfsd4_queue_cb()returns false because it is already queued from the second break, and we ignore that. Nothing retries, and nothing drops the count.nfsd4_shutdown_callback()waits for that count to hit zero before the client can be freed, so teardown blocks there forever. Hitting it needs a write delegation broken by a reader and then a writer, plus a callback that fails. The client itself doesn't have to be broken for that: a short network blip, or a reconnect that resets the backchannel, is enough.Fix
Two upstream cherry-picks, no local changes. Neither is in 6.12.y yet.
NFSD4_CALLBACK_RUNNINGflag on the callback, so a second break on a delegation that is already being recalled returns early instead of queueing the same work item again. In mainline since v6.15.Testing
Two QEMU VMs, same rig: unpatched 6.12.95 and the patched build on 6.12.99. An NFS client in a netns takes a write delegation, traffic to it is dropped, the lease is broken twice (read open then write open), with a kretprobe on
nfsd4_queue_cb.Unpatched leaked on 14 of 14 iterations. Each leak left another task stuck in
nfsd4_shutdown_callback, 15 by the end, and those clients could not be torn down at all.Patched was clean on 23 of 23, no leaks and nothing stuck. Client teardown and NFS stop/start both complete immediately. 38 delegation recall cycles all served in about 1ms, and a mixed soak of clean recalls, double breaks and locks showed every queued callback destroyed with no skew. No new WARNs on either VM.
Confirmed both commits are actually in the patched kernel:
struct nfsd4_callbackgainedcb_flags, andnfsd_break_deleg_cbdisassembles to thetest_and_set_bitgate plus theclear_biton the failed queue path.