NAS-142112 / 25.10.6 / Fix nfsd client teardown hang from leaked callback count (by ixhamza)

[bugclerk]

Problem

Support raised SEE-589 after a customer's NFS server (~87 NFSv4 clients) wedged for the third time. Once it happens nfsd can't expire clients any more, every later client teardown piles up behind the stuck one, and since the stuck task is the laundromat (nfsd's periodic worker that expires dead clients and cleans up NFSv4 state), all of that cleanup stops too. rpc.nfsd won't die on SIGKILL at that point, so only a reboot clears it.

This isn't new in 25.10.4. As per the attached debugs it hit twice on 6.12.15 (Mar 4 and Jun 20) and once on 6.12.91 (Jul 9), with the same stack every time:

INFO: task kworker/u128:11:979462 blocked for more than 120 seconds.
      Tainted: P        W  OE      6.12.15-production+truenas #1
task:kworker/u128:11 state:D stack:0     pid:979462 tgid:979462 ppid:2      flags:0x00004000
Workqueue: nfsd4 laundromat_main [nfsd]
Call Trace:
 <TASK>
 __schedule+0x461/0xa10
 schedule+0x27/0xd0
 nfsd4_shutdown_callback+0xfe/0x140 [nfsd]
 __destroy_client+0x1fa/0x2b0 [nfsd]
 nfs4_process_client_reaplist+0xa9/0x130 [nfsd]
 laundromat_main+0x1f1/0xa10 [nfsd]
 process_one_work+0x180/0x3a0
 worker_thread+0x2da/0x420
 kthread+0xcf/0x100
 ret_from_fork+0x31/0x50

Root cause

NFSv4 hands clients delegations so they can cache a file locally, and the server has to call the client back to take one away. Each of those callbacks takes a count on cl_cb_inflight, and only nfsd41_destroy_cb() drops it again.

There is one callback object per delegation, so if a delegation gets broken twice, both breaks queue the same work item. The first callback RPC fails, nfsd4_cb_release() tries to requeue it for a retry, nfsd4_queue_cb() returns false because it is already queued from the second break, and we ignore that. Nothing retries, and nothing drops the count.

nfsd4_shutdown_callback() waits for that count to hit zero before the client can be freed, so teardown blocks there forever. Hitting it needs a write delegation broken by a reader and then a writer, plus a callback that fails. The client itself doesn't have to be broken for that: a short network blip, or a reconnect that resets the backchannel, is enough.

Fix

Two upstream cherry-picks, no local changes. Neither is in 6.12.y yet.

• 1054e8f adds a NFSD4_CALLBACK_RUNNING flag on the callback, so a second break on a delegation that is already being recalled returns early instead of queueing the same work item again. In mainline since v6.15.
• fa183d59e531 clears that flag when the queue attempt fails. Without it the flag sticks and every later recall on that delegation gets skipped, so the first commit shouldn't go in alone. Queued in nfsd-next with a Fixes: tag against 1054e8f and Cc: stable, so it travels with it.

Testing

Two QEMU VMs, same rig: unpatched 6.12.95 and the patched build on 6.12.99. An NFS client in a netns takes a write delegation, traffic to it is dropped, the lease is broken twice (read open then write open), with a kretprobe on nfsd4_queue_cb.

Unpatched leaked on 14 of 14 iterations. Each leak left another task stuck in nfsd4_shutdown_callback, 15 by the end, and those clients could not be torn down at all.

Patched was clean on 23 of 23, no leaks and nothing stuck. Client teardown and NFS stop/start both complete immediately. 38 delegation recall cycles all served in about 1ms, and a mixed soak of clean recalls, double breaks and locks showed every queued callback destroyed with no skew. No new WARNs on either VM.

Confirmed both commits are actually in the patched kernel: struct nfsd4_callback gained cb_flags, and nfsd_break_deleg_cb disassembles to the test_and_set_bit gate plus the clear_bit on the failed queue path.

Original PR: #333

[bugclerk]

Jira URL: https://ixsystems.atlassian.net/browse/NAS-142112

[bugclerk]

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.