Add support to Apple container in tools - #4444
Open
mribeirodantas wants to merge 17 commits into
Open
Conversation
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: mashehu <mashehu3@gmail.com> Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JulianFlesch <julianflesch@googlemail.com> Co-authored-by: Matthieu Muffato <mm49@sanger.ac.uk> Co-authored-by: yuxinNing <ningyuxin1999@126.com> Co-authored-by: yuxinNing <92061677+ningyuxin1999@users.noreply.github.com> Co-authored-by: JosuaCarl-Agents <josua.carl@web.de> Co-authored-by: CodSpeed Bot <no-reply@codspeed.io> Co-authored-by: louis <louislenezet@gmail.com> Co-authored-by: Louis Le Nézet <58640615+LouisLeNezet@users.noreply.github.com> Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com> Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io> Co-authored-by: Beatriz Vinhas <beatriz.sa.vinhas@gmail.com> Co-authored-by: Nicolas Vannieuwkerke <nicolas.vannieuwkerke@ugent.be> Co-authored-by: Alexander Peltzer <apeltzer@users.noreply.github.com> Co-authored-by: Adrien Coulier <adrien.coulier@pixelgen.com> Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com> Co-authored-by: simojoe <37026683+simojoe@users.noreply.github.com> Co-authored-by: Joël Simoneau <simoneaujoel@gmail.com> Co-authored-by: Vincent Gao <gaobing1230@gmail.com> Co-authored-by: James A. Fellows Yates <jfy133@gmail.com> Co-authored-by: JulianFlesch <julian.flesch@qbic.uni-tuebingen.de> Co-authored-by: znorgaard <zach@fulcrumgenomics.com> Co-authored-by: Josua Carl <53010862+JosuaCarl@users.noreply.github.com> Co-authored-by: sanmaxdev <sanmaxdev@users.noreply.github.com> Co-authored-by: Sangeeth Thilakarathna <46221775+sanmaxdev@users.noreply.github.com> Co-authored-by: codspeed-hq[bot] <117304815+codspeed-hq[bot]@users.noreply.github.com> Co-authored-by: Damon-Lee Pointon <51855558+DLBPointon@users.noreply.github.com> Co-authored-by: Matthias Zepper <6963520+MatthiasZepper@users.noreply.github.com>
- Add template_features.yml toggle for Apple Container in the template features - Add integration tests
Codecov Report❌ Patch coverage is ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Apple Container runs containers only natively on Apple Silicon (arm64) with no Rosetta/QEMU emulation, so it cannot run amd64 biocontainer images. Without an arm64 image source the appleContainer profile stalls trying to use incompatible images. Wire the appleContainer profile to build native arm64 images on demand via Wave from the module's conda packages: process.arch = 'arm64' wave.enabled = true wave.freeze = true wave.strategy = 'conda,container' Applied consistently to: - nf_core/pipeline-template/nextflow.config (generated pipelines) - tests/data/mock_pipeline_containers/nextflow.config (test fixture parity) - template_features.yml (documents behavior) Verified: nf-test on the seqkit/stats module with --profile appleContainer builds a native arm64 image via Wave and passes all tests. Document the limitations in template_features.yml: - Only tools with an arm64 build (conda linux-aarch64 package or multi-arch container image) can be provisioned. amd64-only tools such as prinseq-plus-plus and fastqc cannot run under Apple Container; use the docker profile with amd64 emulation for those. - The profile works without a Seqera token, but wave.freeze pushes anonymous frozen images to the community registry where they expire (causing occasional rebuilds). Setting TOWER_ACCESS_TOKEN is recommended so frozen images are durable and reproducible, after which runs simply pull the cached arm64 image.
Member
Author
|
What's missing for this feature: in |
mribeirodantas
marked this pull request as ready for review
August 21, 2026 03:36
…ntainer profile
The test_containers_pipeline_apple_container test drives `nextflow inspect`
under `-profile appleContainer,test,test_full` to verify container discovery.
The fixture's appleContainer profile mirrored the production template and
enabled Wave freeze (wave.enabled + wave.freeze + wave.strategy =
'conda,container', process.arch = 'arm64').
This made `nextflow inspect` call the live Wave API for every container in the
mock pipeline, which cannot work in CI:
- Freeze mode without a build repository is rejected outright:
POST /v1alpha2/container [400] "Attribute `buildRepository` must be
specified when using freeze mode".
- Even with a build repository, the mock's `container` directives (e.g.
quay.io/biocontainers/singlequay:1.9--pyh9f0ad1d_0) are pushed through
Wave and fail on registry auth:
POST /v1alpha2/container [401] "repository '...' unauthorized".
Either way `nextflow inspect` dies on the Wave call before reaching the
strict-syntax DownloadError the test asserts on, and the test depends on live
Wave + registry auth, making it fragile.
Container discovery via `nextflow inspect` does not need Wave: the fixture only
needs the container system enabled so declared images resolve. This drops all
wave.* settings from the fixture's appleContainer profile so it behaves like
the sibling docker/apptainer/singularity profiles (none of which enable Wave).
The production template profile (nf_core/pipeline-template/nextflow.config) is
intentionally left unchanged; Wave freeze there is a runtime concern for real
arm64 pipelines, not something this discovery test should exercise.
Co-authored-by: mribeirodantas <1023197+mribeirodantas@users.noreply.github.com>
Contributor
Hooray! CodSpeed harness just leveled up!The base and head of this comparison were measured with different runner settings, so their benchmark values are not directly comparable. What changed between base and head:
Re-run the base with the same settings to get a valid performance comparison. Comparing 
Footnotes |
Contributor
|
do you have a working test of this setup in a pipeline? |
Member
Author
Go to your clone of the modules repo and add the appleContainer profile to the Install the PR version of nf-core/tools (I did it with You should see: PS: You need to be on macOS and have If you want to try native arm64 containers, you can use the Wave profile below, like you added the appleContainer (at the beginning of this message): Then The only issue with this one is that there could be a module tagging this one and the tagging module does not have an arm64 conda package, so the building will fail for this one. That's exactly what happens for this example: |
Apple Container runs on Apple Silicon (arm64) but, like Docker, can run amd64 images via emulation with `--platform linux/amd64`. The previous support assumed Apple Container could only run native arm64 and forced every run through Wave to build arm64 images from conda — slow, network-dependent, and broken for tools without an arm64 build. Since most nf-core modules ship amd64 container images, the profile now emulates amd64 by default and works out of the box with no Wave dependency. Native arm64 via Wave becomes an explicit opt-in. - nextflow.config: appleContainer profile now sets `process.arch = 'linux/amd64'` and passes `--platform linux/amd64`; appends `--rosetta` when the new `--apple_rosetta` param is set. Add a composable `appleContainerWave` profile for native arm64 via Wave (`-profile appleContainer,appleContainerWave`). - template_features.yml: add the `apple_rosetta` param (default false, gated behind the apple_container feature) and rewrite the help text to document emulation, Rosetta, and the Wave opt-in. - containers_utils.py: add `appleContainer_amd64` alongside `appleContainer_arm64` so generated offline container configs match both profiles. - download/apple_container.py: make the pull/save `--platform` value configurable via `image_arch`, defaulting to linux/amd64 instead of a hardcoded linux/arm64. - Update mock fixture comment and download tests to expect linux/amd64.
…table profile Previously appleContainerWave was an opt-in override meant only to be composed after appleContainer (`-profile appleContainer,appleContainerWave`). It relied on the base profile to disable the other container engines, so it could not be used on its own — which broke single-profile callers like `nf-core modules/subworkflows test --profile ...`. Make the profile stand alone by explicitly enabling appleContainer and disabling conda/docker/singularity/podman/shifter/charliecloud/apptainer, so it works both standalone (`-profile appleContainerWave`) and composed after appleContainer. Also expose it through the tooling: - Add "appleContainerWave" to the --profile choices for `modules create` and `subworkflows create`. - Add "Apple Container (Wave/arm64)" to the interactive test profile picker and map it to the appleContainerWave Nextflow profile.
Contributor
|
Um, is this an issue with this PR, or a seperate issue in your copied output @mribeirodantas ?
|
pontus
reviewed
Aug 25, 2026
Contributor
|
For clarity so that I understand - apple containers also support arm64 containers and if you want to use (for a process that has a configured What happens on mismatches/if |
Member
Author
Unrelated to the PR. |
Member
Author
Apple container runs arm64 container images by default. If the container image that you picked is not arm64 or multi-arch without arm64 included, it will fail. You can read about it here, in the Nextflow docs. That's why in this PR I set process.arch = 'linux/amd64' by default. If the user wants to pick the arm64 version in multi-arch images, or wants to run arm64 images, they can leave the |
mribeirodantas
commented
Aug 25, 2026
Member
Author
|
LGTM? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds first-class support for Apple container as a container runtime alongside Docker, Singularity, Conda, etc. Apple container runs OCI images natively on Apple Silicon without Docker Desktop.
What's included
Tests & CI
PR checklist
CHANGELOG.mdis updateddocsis updatedCloses #4456