Add WDL implementation for Salmon - #326
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
|
|
||
| runtime { | ||
| cpu: ncpu | ||
| memory: "16 GB" |
There was a problem hiding this comment.
Does salmon use a consistent amount of RAM or is it dependent on the input and/or transcriptome?
There was a problem hiding this comment.
Sorry, I originally hardcoded 16 GB without thinking much about how the memory usage would actually vary.
Salmon's memory usage mainly depends on the index size, which depends on the transcriptome size and whether decoys are included. The input reads are streamed, so their size doesn't have as much impact on RAM usage.
I've now updated both build_salmon_index and quant to calculate memory_gb dynamically based on the input size, similar to how disk_size_gb is already handled. There's also a modify_memory_gb option if the estimate needs to be adjusted for specific data.
| "~{if length(read_twos) == 0 then "--fldMean " + fld_mean else ""}" \ | ||
| "~{if length(read_twos) == 0 then "--fldSD " + fld_sd else ""}" \ |
There was a problem hiding this comment.
I don't think this will work as the arguments end up quoted in bash. Was this an attempt to address a sprocket lint warning?
|
This also needs a |
…d FASTQ data, add SE test
c2d9dde to
4f222d5
Compare
a-frantz
left a comment
There was a problem hiding this comment.
This is looking really great! Left some comments to address, but this is close to the finish line 🚀
| cpu: ncpu | ||
| memory: "~{ceil(transcripts_fasta_size * 4) + 4 + modify_memory_gb} GB" | ||
| disks: "~{disk_size_gb} GB" | ||
| container: "quay.io/biocontainers/salmon:1.9.0--h7e5ed60_0" |
There was a problem hiding this comment.
can we update to a Salmon-v2 container?
There was a problem hiding this comment.
Quick note on the v2 migration: I updated the container and confirmed all our command-line flags remain compatible (both index and quant), and all 4 tests pass. But one thing worth flagging — the new SSHash-based index format is significantly larger than v1's for the same reference: our GRCh38.chrY_chrM.fa-based fixture went from 224MB (v1/pufferfish) to 1.15GB (v2/SSHash). Given the earlier concern about repo/LFS bloat, wanted to surface this before it lands — let me know if you'd still like to proceed with committing this larger fixture, or if you'd prefer a different approach (e.g., a smaller subset reference, or building the index fresh in CI rather than committing it).
There was a problem hiding this comment.
wow, that size explosion is shocking! Pinging @adthrasher as he might find this interesting.
I did some digging this morning, and I think the reason a tiny ref FASTA is creating such a huge index is because we are using subsetted chromosomes from the genome as the reference sequences, where Salmon is expecting individual transcripts from the transcriptome. There is likely an exponential relationship between ref seq length and disk size.
Interestingly, when I tried passing the chr1_chr19 fasta as a decoy to the chrY_chrM build, the resulting index shrank by more than half.
Ultimately though, I think we should commit an appropriate transcriptome reference and use that here, instead of taking the lazy shortcut I suggested earlier 😅 I will create a usable ref FASTA from the human reference transcriptome and commit it to main, then you can rebase this PR and use the file I upload as an input to salmon index for the tests.
There was a problem hiding this comment.
creating- gencode.v50.BCR_ABL1.transcripts.fa.gz — confirmed the index is now genuinely tiny (311KB vs. 1.1GB before), so that fully resolves the size issue!
One thing I've hit: our existing quant tests use the shared fastqs/test_R1.fq.gz/test_R2.fq.gz fixtures, but those were simulated from chrY_chrM — Salmon (correctly) reports 0 fragments mapping against the new BCR/ABL1 transcriptome, since they're from completely different genes. Do you have or know of any existing fixture reads that actually originate from BCR/ABL1, or would it make sense for me to generate a small synthetic FASTQ pair from within gencode.v50.BCR_ABL1.transcripts.fa.gz itself (similar to how I originally built a synthetic transcript, but this time going the other direction — deriving reads from a real transcript)?
| tar -czf "~{prefix}.tar.gz" "~{prefix}" | ||
| >>> | ||
|
|
||
| output { |
There was a problem hiding this comment.
can we also output the "raw" quant.sf file? - https://combine-lab.github.io/salmon/reference/output-formats/
this should be in addition the tarballed output. Some users will likely only care about the quant file, and for running a workflow the quant file may be the only one that's needed downstream so having to extract it from a tarball is a cumbersome intermediate step.
…add decoy-aware indexing test
_Describe the problem or feature in addition to a link to the issues._ This is needed for writing decent tests in #326 BCR and ABL1 were chosen somewhat arbitrarily. We do have other test fixtures referencing these genes specifically (tests are for Arriba), but this doesn't interact with those tests v50 release chosen as its latest. Other fixtures use v31; I could use that release if we want Before submitting this PR, please make sure: - [x] You have added a few sentences describing the PR here. - [x] The code passes all CI tests without any errors or warnings. - [ ] You have added tests (when appropriate). - [ ] You have added an entry in any relevant CHANGELOGs (when appropriate). - [ ] If you have made any changes to the `scripts/` or `docker/` directories, please ensure any image versions have been incremented accordingly! - [x] You have updated the README or other documentation to account for these changes (when appropriate).
Adds a WDL implementation for Salmon (mapping-mode only), per the "tool wishlist" issue #228.
Tasks added in
tools/salmon.wdl:build_salmon_index— wrapssalmon indexquant— wrapssalmon quantAll "important options" from Salmon's docs are exposed as inputs, with defaults verified against
salmon quant --help-readsoutput on Salmon 1.9.0. Parameter documentation is copied from Salmon's official docs, per guidance in the issue. Scoped to mapping-mode only (FASTQ input) — no BAM/alignment-mode support, as requested.Tests added in
tools/salmon.ymlusing the new Sprocket test framework, covering both tasks with real output assertions. Verified locally:sprocket lintpasses cleanly,sprocket dev testpasses both tests.This is my first contribution to this project — happy to make any adjustments you'd like!
Before submitting this PR, please make sure:
scripts/ordocker/directories, please ensure any image versions have been incremented accordingly!