Write TSV, JSON and text files atomically - #62
Merged
rossant merged 2 commits intoAug 9, 2026
Conversation
save_json(), write_python(), write_text(), write_tsv() and _write_tsv_simple() all opened the destination in 'w' mode, which truncates it before a single byte of the new content is written. A crash, a full disk or a kill in that window leaves the file empty or half written and the previous contents are gone. cluster_group.tsv is the one that hurts, since it holds manual curation labels that cannot be recomputed. Add _atomic_open(), a context manager that writes to a temporary file in the destination directory and moves it into place with os.replace() on a clean exit, and route all five writers through it. The temporary file is given the permissions of the file it replaces, or the permissions open() would have used for a new file, since tempfile creates its files with 0600.
adityasingh2400
force-pushed
the
fix-atomic-tsv-json-writes
branch
from
August 9, 2026 11:33
8c261fc to
0703f84
Compare
rossant
approved these changes
Aug 9, 2026
rossant
left a comment
Contributor
There was a problem hiding this comment.
Reviewed after merging current master. The atomic writer now creates sibling files with exclusive open semantics so the OS applies umask without mutating process-global state, preserves existing destination modes, fsyncs before replace, and cleans up on exceptions. Focused tests, binary-mode coverage, flake8, and the full 279-test suite pass locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #60, which fixed the same exposure for
spike_clusters.npyand offered to handle the text writers separately. This is that follow-up. It applies to plain master and does not touch any line #60 touches, so the two can land in either order.save_json(),write_python(),write_text(),write_tsv()and_write_tsv_simple()all open the destination in'w'mode, which truncates it before a single byte of the new content is written. A crash, a full disk or a kill inside that window leaves the file empty or half written, and the previous contents are gone.cluster_group.tsvis the one that hurts, because manual curation labels cannot be recomputed from anything else in the directory.On the question of one helper or several: one helper, and this routes all five writers through it. They differ only in what they write into the handle, the open truncate write close shape is identical, and having
_write_tsv_simple()be crash safe whilewrite_tsv()next to it is not would be a trap for whoever reads this file next. So_atomic_open()is a context manager yielding a file object, writing to a temporary file in the destination directory and moving it into place withos.replace()on a clean exit, removing it and re raising otherwise.save_pickle()is deliberately left alone since it delegates tojoblib.dump.One detail worth flagging.
tempfilecreates its files with mode 0600, so a naive rename would quietly make every saved file private to the user who saved it. That is a real problem for datasets on shared lab storage._atomic_open()therefore gives the temporary file the permissions of the file it replaces, or the permissionsopen()would have given a new file.Tests cover the helper directly, replacement, no leftover temporary file, permissions, and the failure path leaving the previous file byte identical, plus end to end crash injection on
write_tsv,_write_tsv_simpleandsave_json. Against master those crash injection tests leavecluster_group.tsvholding only its header row andtest.jsonholding{"a":, which is the data loss in question.Note on how that was verified: the committed tests import
_atomic_open, so a straight revert only yields an ImportError and proves nothing. I ran the same two behavioural tests without the new import against master's_misc.pyto get a genuine before state.pytest phylibis green at 277, the 273 baseline plus 4.flake8output is byte identical to baseline at 17 errors. I also ran the suite under-W error::ResourceWarningto confirm no leaked file descriptors fromos.fdopen.Once #60 lands,
_save_npy_atomic()inphylib/io/model.pycould be rewritten on top of_atomic_open()withmode='wb'. I left that out here so the two changes stay independent.Disclosure: this change was prepared with AI assistance. I have reviewed and tested it.