Fix numjobs (fork and thread) by gmelikov · Pull Request #1 · rawstor/fio

gmelikov · 2026-05-31T16:45:40Z

No description provided.

Three bugs when fio uses --thread (all jobs share one process): 1. rawstor_initialize() is not idempotent: it asserts io_queue==nullptr, so concurrent calls from multiple threads crash or clobber the global io_queue pointer. 2. rawio::Queue / io_uring SQ+CQ ring is not thread-safe. Concurrent rawstor_object_pread/pwrite (SQE submission) and rawstor_wait() (io_uring_submit_and_wait_timeout) race on the ring state. 3. opened_files counter was used both as a "library initialized" sentinel and as a file count, so rawstor_terminate() was never called when only one file was open (count reached 1, not 0). Fix: introduce a recursive pthread mutex (g_rawstor_mutex) and a reference count (g_rawstor_users) that guard all rawstor API calls. rawstor_acquire/release replace the direct initialize/terminate calls and ensure the library is initialized exactly once across all threads. fio_rawstor_queue wraps pread/pwrite submission under the mutex. fio_rawstor_getevents holds the mutex for the check+wait cycle so that completion flag reads have correct memory ordering and rawstor_wait() is not called concurrently. ETIME from rawstor_wait() is no longer fatal: it means another thread already processed the completions; the loop rechecks io_u_qiter and finds them. In fork mode (default) the mutex is per-process with no contention.

With thread_local io_queue in librawstor, the engine no longer needs a mutex to serialize SQE submissions or CQ waits. Remove the global pthread_mutex and all RAWSTOR_LOCK/UNLOCK wrappers. Store the per-thread RawstorObject* in rawstor_data instead of FILE_ENG_DATA (fio_file->engine_data). With --thread, fio worker threads share the same fio_file struct, so FILE_SET_ENG_DATA from multiple threads races and all threads end up calling pread on the same object concurrently. Keeping the object in the per-thread rawstor_data avoids the race entirely.

gemini-code-assist

Code Review

This pull request refactors the librawstor engine to store the RawstorObject pointer in the per-thread rawstor_data structure and adds timeout handling in fio_rawstor_getevents. However, storing the object pointer in rawstor_data breaks support for multi-file jobs, as fio_file is not shared between threads; the changes should be reverted to use FILE_SET_ENG_DATA and FILE_ENG_DATA. Additionally, continuing the loop on ETIME or ETIMEDOUT in fio_rawstor_getevents can cause an infinite hang, so the engine should return the gathered events instead.

gemini-code-assist · 2026-05-31T16:47:03Z

 struct rawstor_data {
-    int opened_files;
+    int opened_files;   /* number of open rawstor objects on this thread */
    struct io_u **events;
    int queued;
+    /*
+     * Per-thread object pointer.  With fio --thread, multiple threads share
+     * the same fio_file struct, so FILE_ENG_DATA would be a shared write and
+     * all threads would end up pointing at the same RawstorObject.  Storing
+     * the object here (in the per-thread rawstor_data) avoids the race.
+     * Assumes one file per job, which is the standard rawstor use-case.
+     */
+    RawstorObject *object;
 };


Storing a single RawstorObject pointer in the per-thread rawstor_data struct (rd->object) breaks support for jobs with multiple files (nrfiles > 1).

In fio, struct fio_file is not shared between threads/jobs, even when using --thread (each thread gets its own cloned fio_file structures via dup_files). Therefore, using FILE_SET_ENG_DATA and FILE_ENG_DATA is completely thread-safe and is the standard, idiomatic way to store per-file engine data in fio.

By switching to rd->object:

Opening multiple files will overwrite rd->object with the last opened file's object.

All queued I/O operations will be directed to the last opened file's object, ignoring the actual target file of the io_u.

Closing files will call rawstor_object_close on the same object multiple times (leading to potential double-free/crashes), while leaking all other opened objects.

We should revert this change and continue using FILE_SET_ENG_DATA and FILE_ENG_DATA.

struct rawstor_data { int opened_files; /* number of open rawstor objects on this thread */ struct io_u **events; int queued; };

gemini-code-assist · 2026-05-31T16:47:03Z

 {
    struct rawstor_data *rd = td->io_ops_data;
-    RawstorObject *object = FILE_ENG_DATA(io_u->file);
+    RawstorObject *object = rd->object;


Revert to using FILE_ENG_DATA(io_u->file) to retrieve the correct RawstorObject for the target file, ensuring multi-file jobs work correctly.

RawstorObject *object = FILE_ENG_DATA(io_u->file);

gemini-code-assist · 2026-05-31T16:47:03Z


    ++rd->opened_files;
-    FILE_SET_ENG_DATA(f, object);
+    rd->object = object;


Revert to using FILE_SET_ENG_DATA(f, object) to store the RawstorObject per-file, ensuring multi-file jobs work correctly.

FILE_SET_ENG_DATA(f, object);

gemini-code-assist · 2026-05-31T16:47:03Z

 {
    struct rawstor_data *rd = td->io_ops_data;
-    RawstorObject *object = FILE_ENG_DATA(f);
+    RawstorObject *object = rd->object;


Revert to using FILE_ENG_DATA(f) to retrieve the correct RawstorObject for the file being closed, ensuring multi-file jobs work correctly.

RawstorObject *object = FILE_ENG_DATA(f);

gemini-code-assist · 2026-05-31T16:47:03Z

+            if (-res == ETIME || -res == ETIMEDOUT)
+                continue;


Using continue when rawstor_wait() returns -ETIME or -ETIMEDOUT can cause an infinite loop / hang in fio_rawstor_getevents if the requested min number of events is not met and the I/O is delayed or lost.

Fio's getevents is expected to respect timeouts. If a timeout occurs, the engine should return the events gathered so far (even if less than min), allowing the fio control loop to run, handle signals, check job limits, etc. Continuing indefinitely will block the thread forever in case of a slow or stuck backend.

We should return events instead of continuing.

if (-res == ETIME || -res == ETIMEDOUT) return events;

gmelikov added 2 commits May 31, 2026 16:43

gemini-code-assist Bot reviewed May 31, 2026

View reviewed changes

VasilyStepanov force-pushed the rawstor branch 3 times, most recently from 662672b to c27597e Compare June 3, 2026 14:36

Navigation Menu

Uh oh!

Fix numjobs (fork and thread) - #1

Fix numjobs (fork and thread)#1
gmelikov wants to merge 2 commits into
rawstorfrom
multibackend

gmelikov commented May 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Uh oh!

gemini-code-assist Bot May 31, 2026

Uh oh!

gemini-code-assist Bot May 31, 2026

Uh oh!

gemini-code-assist Bot May 31, 2026

Uh oh!

gemini-code-assist Bot May 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

gmelikov commented May 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant