Skip to content

Write TSV, JSON and text files atomically - #62

Merged
rossant merged 2 commits into
cortex-lab:masterfrom
adityasingh2400:fix-atomic-tsv-json-writes
Aug 9, 2026
Merged

Write TSV, JSON and text files atomically#62
rossant merged 2 commits into
cortex-lab:masterfrom
adityasingh2400:fix-atomic-tsv-json-writes

Conversation

@adityasingh2400

Copy link
Copy Markdown
Contributor

Follow-up to #60, which fixed the same exposure for spike_clusters.npy and offered to handle the text writers separately. This is that follow-up. It applies to plain master and does not touch any line #60 touches, so the two can land in either order.

save_json(), write_python(), write_text(), write_tsv() and _write_tsv_simple() all open the destination in 'w' mode, which truncates it before a single byte of the new content is written. A crash, a full disk or a kill inside that window leaves the file empty or half written, and the previous contents are gone. cluster_group.tsv is the one that hurts, because manual curation labels cannot be recomputed from anything else in the directory.

On the question of one helper or several: one helper, and this routes all five writers through it. They differ only in what they write into the handle, the open truncate write close shape is identical, and having _write_tsv_simple() be crash safe while write_tsv() next to it is not would be a trap for whoever reads this file next. So _atomic_open() is a context manager yielding a file object, writing to a temporary file in the destination directory and moving it into place with os.replace() on a clean exit, removing it and re raising otherwise. save_pickle() is deliberately left alone since it delegates to joblib.dump.

One detail worth flagging. tempfile creates its files with mode 0600, so a naive rename would quietly make every saved file private to the user who saved it. That is a real problem for datasets on shared lab storage. _atomic_open() therefore gives the temporary file the permissions of the file it replaces, or the permissions open() would have given a new file.

Tests cover the helper directly, replacement, no leftover temporary file, permissions, and the failure path leaving the previous file byte identical, plus end to end crash injection on write_tsv, _write_tsv_simple and save_json. Against master those crash injection tests leave cluster_group.tsv holding only its header row and test.json holding {"a":, which is the data loss in question.

Note on how that was verified: the committed tests import _atomic_open, so a straight revert only yields an ImportError and proves nothing. I ran the same two behavioural tests without the new import against master's _misc.py to get a genuine before state. pytest phylib is green at 277, the 273 baseline plus 4. flake8 output is byte identical to baseline at 17 errors. I also ran the suite under -W error::ResourceWarning to confirm no leaked file descriptors from os.fdopen.

Once #60 lands, _save_npy_atomic() in phylib/io/model.py could be rewritten on top of _atomic_open() with mode='wb'. I left that out here so the two changes stay independent.

Disclosure: this change was prepared with AI assistance. I have reviewed and tested it.

save_json(), write_python(), write_text(), write_tsv() and _write_tsv_simple()
all opened the destination in 'w' mode, which truncates it before a single byte
of the new content is written. A crash, a full disk or a kill in that window
leaves the file empty or half written and the previous contents are gone.

cluster_group.tsv is the one that hurts, since it holds manual curation labels
that cannot be recomputed.

Add _atomic_open(), a context manager that writes to a temporary file in the
destination directory and moves it into place with os.replace() on a clean
exit, and route all five writers through it. The temporary file is given the
permissions of the file it replaces, or the permissions open() would have used
for a new file, since tempfile creates its files with 0600.
@adityasingh2400
adityasingh2400 force-pushed the fix-atomic-tsv-json-writes branch from 8c261fc to 0703f84 Compare August 9, 2026 11:33

@rossant rossant left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed after merging current master. The atomic writer now creates sibling files with exclusive open semantics so the OS applies umask without mutating process-global state, preserves existing destination modes, fsyncs before replace, and cleans up on exceptions. Focused tests, binary-mode coverage, flake8, and the full 279-test suite pass locally.

@rossant
rossant merged commit 0f78527 into cortex-lab:master Aug 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants