Repository navigation
Speed up upload-to-s3 #41
Description
Activity
Update hashing to stop going through Python.
This is currently blocked on
sha256sumnot being available in the conda runtime.Alternatively, the naive
sha256sumimplementation in Python could exec into the GNU coreutils version, if found, otherwise fall back to the slow Python version.Reacted by Jover Lee- added a commit that references this issue
on Jun 11, 2024 Update hashing to stop going through Python.
This is currently blocked on
sha256sumnot being available in the conda runtime.sha256sumis now available as of the nextstrain-base 20240612T205814Z Conda package.Note that we're (I'm) assuming the GNU coreutils
sha256sumimplementation is faster than our Python one. It likely is! But we don't actually know. Benchmarking might be useful here.Reacted by Jover LeeNote that we're (I'm) assuming the GNU coreutils sha256sum implementation is faster than our Python one. It likely is! But we don't actually know. Benchmarking might be useful here.
Simple test with a 1.3G fasta file.
$ time sha256sum ./ingest/data/ncbi_dataset_sequences.fasta f85d4bfc6c9cfc00d10567aed87723c4bf39498b5dc94f81c94f4b31c98fb806 ./ingest/data/ncbi_dataset_sequences.fasta real 0m4.452s user 0m3.952s sys 0m0.483s$ time ./ingest/vendored/sha256sum ./ingest/data/ncbi_dataset_sequences.fastaStill running after 10mins...I'll post update with final time after it finishes...
Reacted by Jennifer ChangStill running after 10mins...I'll post update with final time after it finishes...
🤦♀️ Nope, I was just running the script wrong
$ time ./ingest/vendored/sha256sum < ./ingest/data/ncbi_dataset_sequences.fasta f85d4bfc6c9cfc00d10567aed87723c4bf39498b5dc94f81c94f4b31c98fb806 real 0m1.401s user 0m0.539s sys 0m0.841sWait, is the Python one actually faster? What? I mean, I know the hashlib implementations are in C as is much file i/o, but I still would expect Python overhead to be significant here.
Is your coreutils
sha256sumx86_64 or aarch64?file $(type -p sha256sum)Is your coreutils sha256sum x86_64 or aarch64? file $(type -p sha256sum)
Ah, should have said I was running these in the Nextstrain shell using the Docker runtime.
One thing I noted looking at coreutils sha256sum is it's reading in 32 kiB chunks vs. our 5 MiB chunks.
Similar results when running in macOS terminal:
KX76YWH7NX:mpox joverlee$ time sha256sum ingest/data/ncbi_dataset_sequences.fasta f85d4bfc6c9cfc00d10567aed87723c4bf39498b5dc94f81c94f4b31c98fb806 ingest/data/ncbi_dataset_sequences.fasta real 0m5.059s user 0m4.725s sys 0m0.209s KX76YWH7NX:mpox joverlee$ time ./ingest/vendored/sha256sum < ingest/data/ncbi_dataset_sequences.fasta f85d4bfc6c9cfc00d10567aed87723c4bf39498b5dc94f81c94f4b31c98fb806 real 0m1.439s user 0m0.632s sys 0m0.221s KX76YWH7NX:mpox joverlee$ file $(type -p sha256sum) /opt/homebrew/bin/sha256sum: Mach-O 64-bit executable arm64Just making sure this is true for a larger file, testing with a 70G fasta. Python is much faster than GNU coreutils!
GNU coreutils:
KX76YWH7NX:ncov-ingest joverlee$ time sha256sum data/gisaid/sequences.fasta 3f47c5c48118ec5da9955bffe4346ea6245ad6d6b443c544f85c7f4d377a4b1e data/gisaid/sequences.fasta real 4m16.720s user 3m59.214s sys 0m8.749sPython:
KX76YWH7NX:ncov-ingest joverlee$ time ./vendored/sha256sum < data/gisaid/sequences.fasta 3f47c5c48118ec5da9955bffe4346ea6245ad6d6b443c544f85c7f4d377a4b1e real 0m45.221s user 0m30.535s sys 0m6.089sWow. I'd be really curious what the times are if you drop our read size in Python to 32 kiB.
I'd also wonder if aarch64 is coming into play here: is Python taking advantage of it (and coreutils not) in a way it couldn't on x86_64 hardware we're using on AWS Batch?
On my machine, Python is only slightly faster than coreutils. In fact, alternative non-cryptographic/secure hashing algorithms I've tried (a few impls of MurmurHash3, simple crc32, simple md5) all come out very roughly in the same ballpark (within ~20s of each other on a 3GB file), which leads me to thinking I'm bottlenecking on i/o on my machine. And so I'd wonder if we hit an i/o bottleneck in Batch too. We're not using fast disks on AWS...
I'd be really curious what the times are if you drop our read size in Python to 32 kiB.
It's actually slightly faster when I drop the chunk size
KX76YWH7NX:ncov-ingest joverlee$ time ./vendored/sha256sum < data/gisaid/sequences.fasta chunk size: 32768 3f47c5c48118ec5da9955bffe4346ea6245ad6d6b443c544f85c7f4d377a4b1e real 0m41.200s user 0m30.406s sys 0m8.086s- linked a pull request that will close this issueupload-to-s3: read the source once instead of three times #77
on Jul 2, 2026 My brief testing on a 24 GB chunk of
aligned.fastashowed pretty equivalent performance between Python and coreutilssha256sum.┌─────────────────────────────┬────────┬────────────┐ │ phase │ time │ throughput │ ├─────────────────────────────┼────────┼────────────┤ │ cat (raw-read floor) │ 101.1s │ 236 MB/s │ ├─────────────────────────────┼────────┼────────────┤ │ vendored sha256sum (python) │ 95.3s │ 250 MB/s │ ├─────────────────────────────┼────────┼────────────┤ │ coreutils sha256sum │ 104.3s │ 229 MB/s │ ├─────────────────────────────┼────────┼────────────┤ │ wc -l │ 94.3s │ 253 MB/s │ ├─────────────────────────────┼────────┼────────────┤ │ zstd -T0 -c │ 99.2s │ 240 MB/s │ └─────────────────────────────┴────────┴────────────┘The slowness was no more
sha256sumthanwc -lor compression.
Prompted by nextstrain/ncov-ingest#446
Some ideas of speeding up
upload-to-s3proposed in related Slack thread