nvme-cli-3.0-b.2 (git 3.0-b.2): Segfault in libnvme3.so during parallel nvme discover loop execution

[adhipashelly]

CONFIGURATION

• NVMe Over Fabrics products.

STEPS TO REPRODUCE

1. Run nvme discover command in parallel on 24 drives in loop.

OBSEVATION

• Observing segmentaion fault under dmesg logs related to libnvme.

DMESG LOGS
[Fri Jul 10 06:31:28 2026] nvme[933315]: segfault at 8 ip 00007f99756bdddf sp 00007ffd0cad95a0 error 4 in libnvme3.so.3.0.0[7f99756ba000+24000]
[Fri Jul 10 06:31:28 2026] Code: 44 89 53 24 4c 8d 5c 05 00 4c 01 fd 4c 89 6b 18 4c 39 e5 4c 89 5b 30 0f 92 c0 0a 44 24 1b 0b 74 24 10 0f b6 c0 c1 e0 0f 09 f0 <41> 83 7e 08 01 89 43 28 75 57 41 83 be d0 00 00 00 01 74 4d 31 d2
[Fri Jul 10 06:31:28 2026] nvme nvme56: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:28 2026] nvme nvme56: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme0: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme51: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme51: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme54: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme54: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme54: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:29 2026] nvme nvme55: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme55: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme57: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme57: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme55: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:29 2026] nvme nvme54: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme54: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme57: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:29 2026] nvme nvme59: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:29 2026] nvme nvme59: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:29 2026] nvme nvme54: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:29 2026] nvme nvme55: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme55: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme53: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme57: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme57: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme59: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme54: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme54: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme55: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme53: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme53: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme60: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme60: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme53: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme59: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme59: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme60: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme55: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme55: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme52: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:30 2026] nvme nvme53: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:30 2026] nvme nvme53: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:30 2026] nvme nvme59: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme52: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:31 2026] nvme nvme52: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:31 2026] nvme nvme55: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme59: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:31 2026] nvme nvme58: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme59: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:31 2026] nvme nvme53: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme52: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme55: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:31 2026] nvme nvme55: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:31 2026] nvme nvme56: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme59: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:31 2026] nvme nvme58: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:31 2026] nvme nvme58: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme55: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme52: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme52: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme53: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme53: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme56: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme[933810]: segfault at 8 ip 00007f9773c2eddf sp 00007fff4532d6a0 error 4 in libnvme3.so.3.0.0[7f9773c2b000+24000]
[Fri Jul 10 06:31:32 2026] Code: 44 89 53 24 4c 8d 5c 05 00 4c 01 fd 4c 89 6b 18 4c 39 e5 4c 89 5b 30 0f 92 c0 0a 44 24 1b 0b 74 24 10 0f b6 c0 c1 e0 0f 09 f0 <41> 83 7e 08 01 89 43 28 75 57 41 83 be d0 00 00 00 01 74 4d 31 d2
[Fri Jul 10 06:31:32 2026] nvme nvme56: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.10.10.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme58: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme0: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme55: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme55: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme59: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme59: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme55: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme60: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme51: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme60: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme61: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme61: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme52: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme[933905]: segfault at 8 ip 00007f2f1f781ddf sp 00007ffcdab341d0 error 4 in libnvme3.so.3.0.0[7f2f1f77e000+24000]
[Fri Jul 10 06:31:32 2026] Code: 44 89 53 24 4c 8d 5c 05 00 4c 01 fd 4c 89 6b 18 4c 39 e5 4c 89 5b 30 0f 92 c0 0a 44 24 1b 0b 74 24 10 0f b6 c0 c1 e0 0f 09 f0 <41> 83 7e 08 01 89 43 28 75 57 41 83 be d0 00 00 00 01 74 4d 31 d2
[Fri Jul 10 06:31:32 2026] nvme[933921]: segfault at 8 ip 00007f92a553cddf sp 00007fff21ad7f00 error 4 in libnvme3.so.3.0.0[7f92a5539000+24000]
[Fri Jul 10 06:31:32 2026] Code: 44 89 53 24 4c 8d 5c 05 00 4c 01 fd 4c 89 6b 18 4c 39 e5 4c 89 5b 30 0f 92 c0 0a 44 24 1b 0b 74 24 10 0f b6 c0 c1 e0 0f 09 f0 <41> 83 7e 08 01 89 43 28 75 57 41 83 be d0 00 00 00 01 74 4d 31 d2
[Fri Jul 10 06:31:32 2026] nvme nvme58: queue_size 128 > ctrl sqsize 16, clamping down
[Fri Jul 10 06:31:32 2026] nvme nvme58: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 20.20.20.3:4420
[Fri Jul 10 06:31:32 2026] nvme nvme61: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme56: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme57: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme nvme54: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fri Jul 10 06:31:32 2026] nvme[934019]: segfault at 8 ip 00007fd855a8addf sp 00007ffc24147060 error 4 in libnvme3.so.3.0.0[7fd855a87000+24000]
[Fri Jul 10 06:31:32 2026] Code: 44 89 53 24 4c 8d 5c 05 00 4c 01 fd 4c 89 6b 18 4c 39 e5 4c 89 5b 30 0f 92 c0 0a 44 24 1b 0b 74 24 10 0f b6 c0 c1 e0 0f 09 f0 <41> 83 7e 08 01 89 43 28 75 57 41 83 be d0 00 00 00 01 74 4d 31 d2

NVME VERSION
:~# nvme version
nvme version 3.0-b.2 (git 3.0-b.2)
libnvme version 3.0-b.2 (git 3.0-b.2)

[igaw]

Could you either run it under gdb and create a backtrace with something like:

gdb --args nvme discover ...
(gdb) run
(gdb) bt full

Or in case you got a core dump:

gdb /usr/bin/nvme core
(gdb) bt full

Or if you have neither

addr2line -e /usr/lib64/libnvme3.so.3.0.0 0x3ddf

the library needs to built with debug symbols.

[adhipashelly]

Hi @igaw
Please find the requested data - Segfault in libnvme3.so during parallel nvme discover loop execution

gdb_nvme_crash.txt

[igaw]

Thanks a lot.

libnvme_get_log (hdl=hdl@entry=0x0, cmd=cmd@entry=0x7ffeee34dbe0, rae=rae@entry=false, xfer_len=xfer_len@entry=20)

libnvme_get_log is called with NULL pointer for the handle. That's not good.

[igaw]

The problem seems to be that the discover ctrl has been created by one process (this one will have a valid hdl) and another process will find the ctrl in sysfs and wont create a hdl for this one.

[igaw]

Any chance to test PR #3581?