Podman port - #206
Open
rountree wants to merge 20 commits into
Open
Conversation
Revisiting the podman port PR with a more capable LLM. Goal is to avoid the severe resource contention at GitHub Actions. In Claude's words: Add podman hello-world test (01-basic) Introduces initial podman support for running Spindle containers locally on LC systems. This is the first of four progressive hello-world tests that validate the podman environment. 01-basic tests: - apt-get functionality (with LC setgroups workaround) - Container build and execution - Network connectivity - SSL certificate configuration Key features: - Dockerfiles use ARG PODMAN_BUILD for conditional LC fixes - Helper scripts in scripts/podman/ handle LC-specific mounts - Compatible with both Docker (GitHub) and podman (LC) Tested successfully on LC system with podman 4.9.4-rhel. Next steps: 02-user-switch, 03-filesystem, 04-networking
Prior attempts got hung up in creating new users in the container. Got ahead of that this time around. Works fine. Per Claude: Add podman hello-world test (02-user-switch) Implements the second hello-world test validating non-root user patterns used throughout Spindle containers. 02-user-switch tests: - Creating non-root user (uid=1001, gid=1001) - USER directive for switching from root - Home directory setup with correct ownership - File permission handling (read root files, write user dirs) - Sudo access for container operations This pattern is critical for Spindle containers because: - HPC systems run jobs as non-root users - Builds must not create root-owned artifacts - Tests need realistic permission scenarios Tested successfully with podman 4.9.4-rhel on LC system. All permission checks pass as expected. Updated: - containers/spindle-hello-podman/README.md (marked 02 available) - scripts/podman/README.md (updated status) Next: 03-filesystem (volume mounts), 04-networking (multi-container), then 05-flux-multi and 06-slurm-multi
Pulling logs out of the container should now be straightforward.
Per Claude:
Add podman hello-world test (03-filesystem)
Implements the third hello-world test validating volume mount
patterns and filesystem operations between host and container.
03-filesystem tests:
- Volume mounts with -v flag and :Z for SELinux
- User namespace mapping with --userns=keep-id
- Reading files from host-mounted directories
- Writing files to mounted volumes
- File persistence after container exit
- Proper UID handling without world-writable directories
Key implementation details:
- Uses --userns=keep-id to map host UID into container
- Avoids permission issues without compromising security
- ENV USER=${USER} makes build-time ARG available at runtime
- Creates temporary test directories on host
- Verifies artifacts persist after container cleanup
This pattern is critical for Spindle containers:
- Source code mounted from host (no copy into image)
- Build artifacts written to persistent volumes
- Test logs accessible on host after container exits
- Proper ownership: files owned by host user, not root
Tested successfully with podman 4.9.4-rhel on LC system.
All volume operations work correctly with appropriate permissions.
Updated:
- containers/spindle-hello-podman/README.md (marked 03 available)
- scripts/podman/README.md (updated status)
Next: 04-networking (multi-container communication)
Per Claude: Add podman hello-world test (04-networking) Implements the fourth hello-world test validating multi-container networking and service discovery patterns needed for Slurm/Flux clusters. 04-networking tests: - Custom network creation (podman network create) - Multiple containers on the same network - DNS resolution between containers (hostname lookups) - Container-to-container HTTP communication - Service discovery without hard-coded IPs - Automatic cleanup with trap on exit Test architecture: - One Docker image used for both server and client roles - Server: runs simple HTTP service on port 8080 - Client 1: validates DNS resolution (ping by hostname) - Client 2: validates HTTP communication (curl by hostname) - All containers communicate via custom network Key patterns demonstrated: - Single image, multiple roles (command override) - Hostname-based service discovery (not IP-based) - Proper cleanup even on failure (trap EXIT) - Network isolation from host This completes the foundational hello-world series. These patterns are essential for multi-node Slurm/Flux clusters where: - Head nodes must communicate with compute nodes - Service discovery happens by hostname - Multiple containers coordinate as a cluster Tested successfully with podman 4.9.4-rhel on LC system. All containers communicate correctly via custom network. Updated: - containers/spindle-hello-podman/README.md (marked 04 available) - scripts/podman/README.md (updated status) Next: 05-flux-multi (Flux cluster), 06-slurm-multi (Slurm cluster)
Per Claude: Add podman hello-world test (05-flux-multi) Implements a simplified multi-node Flux cluster demonstration using the official fluxrm/flux-sched base image. 05-flux-multi tests: - Official Flux Framework base image (fluxrm/flux-sched:noble-v0.48.0-amd64) - Multi-node cluster: 1 head node + 2 worker nodes - Munge authentication between nodes - Flux broker connections and coordination - Distributed job execution (flux run -N 3 hostname) - Resource management across cluster Architecture: - Single Docker image for all nodes (head + workers) - Behavior differs based on hostname (flux-node-1 is head) - Head node starts broker and waits for workers - Worker nodes connect to head node after 5s delay - Test job runs across all nodes to verify cluster Key patterns demonstrated: - Using official Flux images as base (not building from source) - Flux configuration in /etc/flux/config/broker.toml - Resource encoding with flux R encode - Munge setup for secure inter-node auth - Head vs worker coordination via entrypoint logic This mirrors the pattern used in Spindle's Flux containers, simplified to show just Flux itself working without Spindle. Implementation notes: - COPY and chmod performed as root, then chown to fluxuser - Avoids permission issues with entrypoint script - LC-specific apt fix applied to base image - Uses --userns=keep-id not needed (official image handles UIDs) Tested successfully with podman 4.9.4-rhel on LC system. Flux cluster initializes correctly and runs distributed jobs. Next: 06-slurm-multi (Slurm cluster example), then port actual Spindle serial/flux/slurm containers
Per Claude:
Add 06-slurm-multi: Multi-node Slurm cluster demo
Completes the hello-world series with a working Slurm cluster example.
This was significantly harder than the Flux example (05) due to Slurm's
complexity and rootless podman constraints.
Three major issues solved:
1. **Package installation failure in rootless podman**
Initial approach used Ubuntu packages (slurm-wlm, slurmd, slurmctld).
Failed with "chown: changing ownership of '/var/lib/slurm': Invalid argument"
during dpkg post-install scripts.
Solution: Build Slurm from source instead of using packages, following
Spindle's proven pattern in containers/spindle-slurm-ubuntu/base/.
This completely avoids the dpkg post-install chown issues that occur
in rootless podman's user namespace mapping.
2. **SSL certificate verification during git clone**
Building from source requires cloning from GitHub, which failed with
"server certificate verification failed" on LC systems.
Solution: Add ca-certificates package. Combined with the certificate
volume mounts in common.sh (LC_CERT_BUILD_MOUNT), this provides the
necessary SSL trust chain.
3. **slurmd crashes with cgroup v2 plugin missing**
Worker nodes failed to start with "cannot find cgroup plugin for cgroup/v2"
causing slurmd initialization to fail. Controller couldn't reach workers
because slurmd never started listening on port 6818.
Solution: Add cgroup.conf forcing cgroup/v1, matching Spindle's config.
The cgroup v2 plugin isn't available in this build configuration.
Configuration based on Spindle's working setup:
- Role names: ctl/worker (not controller/compute)
- slurmd runs as root via sudo (needed for cgroup management)
- Explicit NodeAddr for each node in slurm.conf
- ControlMachine instead of SlurmctldHost
- Diagnostic ping/munge tests in worker entrypoint
Files:
- containers/spindle-hello-podman/06-slurm-multi/Dockerfile
- containers/spindle-hello-podman/06-slurm-multi/slurm.conf
- containers/spindle-hello-podman/06-slurm-multi/cgroup.conf (NEW)
- containers/spindle-hello-podman/06-slurm-multi/entrypoint.sh
- scripts/podman/run-hello-06-slurm.sh
Test validates: Slurm build from source, multi-node cluster, munge auth,
node registration, and job submission with srun.
All hello-world tests (01-06) now pass. Ready to port real Spindle containers.
Per Claude:
Add Spindle serial container for podman
Ports the Spindle serial test container to podman with LC-specific fixes.
This is the first real Spindle container (not hello-world demos).
Key changes from Docker version:
1. **LC podman fixes applied**
- PODMAN_BUILD arg for setgroups workaround
- Certificate volume mounts via common.sh
- All patterns validated in hello-world series
2. **Permission handling for COPY operations**
Multiple USER/root switches needed to handle file permissions:
- Spindle repo copied as user, then fixed by root (configure needs +x)
- build_spindle.sh copied and chmod'd as root, then chown'd to user
- entrypoint.sh.podman copied and chmod'd as root
Pattern: COPY as root, chmod/chown as root, then USER switch for execution
3. **UCX warning suppression**
Added ENV UCX_LOG_LEVEL=error to suppress rootless container warnings:
"unable to read somaxconn value from /proc/sys/net/core/somaxconn"
This warning was causing runTests to report false failures. UCX falls
back to defaults; the warning is cosmetic in rootless containers.
4. **Container lifecycle fix**
Original entrypoint ran munged --foreground, which Docker handles but
podman does not. New entrypoint.sh.podman starts munged in background
then runs 'sleep inf' to keep container alive for exec commands.
5. **Test script organization**
- build-spindle-serial.sh: Build the container
- test-spindle-serial.sh: Run regular tests only
- test-spindle-serial-crash.sh: Run crash tests only
Crash tests take significant time; splitting allows quick regular test runs.
Files:
- containers/spindle-serial-ubuntu/Dockerfile.podman (NEW)
- containers/spindle-serial-ubuntu/scripts/entrypoint.sh.podman (NEW)
- scripts/podman/build-spindle-serial.sh (NEW)
- scripts/podman/test-spindle-serial.sh (NEW)
- scripts/podman/test-spindle-serial-crash.sh (NEW)
Test results: All regular tests pass. Crash tests pass.
Next: Port Flux and Slurm Spindle containers.
Single-test only. Per Claude: Add debug logging support for Spindle serial tests Adds script to run tests with SPINDLE_DEBUG=3 and extract logs from container. Key findings: - Spindle log files: spindle_output.<nodename>.<pid> (no .log suffix) - Logs written to test working directory (Spindle-build/testsuite/) - Extraction via podman cp works reliably - Container left running for interactive inspection Script demonstrates pattern for Flux/Slurm containers: 1. Run test with SPINDLE_DEBUG=3 2. Locate spindle_output.* files 3. Extract to host with podman cp 4. Leave container running for further inspection Files: - scripts/podman/test-spindle-serial-debug.sh (NEW) Next: Port Flux and Slurm containers, replicate this debug pattern.
Per Claude:
Ports the Spindle Flux multi-node test container to podman with LC-specific
fixes. Build succeeds and 4-node Flux cluster initializes correctly, but
Spindle tests fail at runtime.
Status:
✓ Container builds successfully
✓ 4-node Flux cluster starts and all nodes register
✓ Munge authentication works
✗ Spindle tests fail with "spindleRunBE failed!" error
✗ No spindle_output.* logs generated (fails before logging starts)
Key changes from Docker version:
1. **LC podman fixes applied**
- PODMAN_BUILD arg for setgroups workaround
- Certificate volume mounts via common.sh
- UCX_LOG_LEVEL=error to suppress warnings
2. **Permission handling for COPY operations**
Same pattern as serial container:
- Spindle repo copied, then permissions fixed as root
- Build script copied and chmod'd as root
- Scripts copied and chmod'd as root
- Added chmod -R a+rX /etc/flux to make configs readable by fluxuser
3. **Podman-specific entrypoint**
- Uses exec before flux start to keep container alive
- Added debug output (set -x, echo statements)
- Starts munged in background before flux broker
4. **Multi-node orchestration**
- test-spindle-flux.sh creates network and starts 4 nodes
- Passes mainHost and workers environment variables
- Checks container health before running tests
- Shows logs on failure
5. **Enhanced common.sh**
- Added extra_args parameter to podman_build() for --build-arg
Files:
- containers/spindle-flux-ubuntu/Dockerfile.podman (NEW)
- containers/spindle-flux-ubuntu/scripts/entrypoint.sh.podman (NEW)
- scripts/podman/build-spindle-flux.sh (NEW)
- scripts/podman/test-spindle-flux.sh (NEW)
- scripts/podman/common.sh (MODIFIED - extra build args support)
Known issues:
- Build has 10M+ page faults (7+ minutes, performance issue)
- spindleRunBE fails immediately on first test
- Needs debugging with SPINDLE_DEBUG=3 on exclusive compute node
- May be podman-specific namespace/permission issue
Flux cluster verification:
- flux resource list shows all 4 nodes allocated
- flux dmesg shows clean startup, no errors
- All containers healthy and running
Next: Port Slurm container, then debug Flux/multiple-commpaths issues.
Per Claude:
Add Spindle Slurm srun container for podman (tests passing!)
Ports the Spindle Slurm srun multi-node test container to podman with LC-specific
fixes. Two-stage build (base + testing). All tests pass successfully!
Cluster: 1 MariaDB + 1 slurmdbd + 1 slurmctld + 4 workers = 7 containers
Status:
✓ Base image builds (Slurm + MPICH from source, 6+ minutes)
✓ Testing image builds (Spindle + config)
✓ 7-container cluster starts and all daemons connect
✓ All Spindle tests pass: "ALL TESTS PASSED"
Key fixes for rootless podman:
1. **MPICH tarball extraction**
- Added --no-same-owner to tar command in build_mpich.sh.podman
- Without this, tar fails with "Cannot change ownership to uid 3328" errors
- Rootless podman can't preserve ownership from tarballs
2. **MariaDB password authentication**
- generate_config.sh creates random password in mariadb.env and slurmdbd.conf
- Test script now reads password from mariadb.env instead of hardcoding
- Fixed: "Access denied for user 'slurm'" preventing slurmdbd connection
3. **Slurm config file permissions**
- setup_slurm.sh.podman makes slurm.conf world-readable (chmod 644)
- slurmdbd.conf stays restricted (chmod 600, has password)
- Fixed: "Permission denied" when slurmuser runs salloc
4. **Task affinity disabled**
- Created slurm.conf.podman with TaskPlugin=task/none
- Original slurm.conf unchanged (Docker still uses task/affinity)
- task/affinity fails with "Operation not permitted" in rootless podman
- CPU affinity requires cgroup controllers not available in user namespaces
- Fixed: "task_g_set_affinity: Operation not permitted" on every task launch
5. **Script permissions after COPY**
- Added chmod +x for build_slurm.sh and build_mpich.sh
- COPY doesn't preserve execute bits
6. **Image name prefix handling**
- Fixed base image check to accept localhost/ prefix
- Podman images are tagged as localhost/spindle-slurm-base
Files:
- containers/spindle-slurm-ubuntu/base/Dockerfile.podman (NEW)
- containers/spindle-slurm-ubuntu/base/scripts/build_mpich.sh.podman (NEW)
- containers/spindle-slurm-ubuntu/testing-srun/Dockerfile.podman (NEW)
- containers/spindle-slurm-ubuntu/testing-srun/conf/slurm.conf.podman (NEW)
- containers/spindle-slurm-ubuntu/testing-srun/scripts/setup_slurm.sh.podman (NEW)
- containers/spindle-slurm-ubuntu/testing-srun/scripts/entrypoint.sh.podman (NEW)
- scripts/podman/build-spindle-slurm-base.sh (NEW)
- scripts/podman/build-spindle-slurm-srun.sh (NEW)
- scripts/podman/test-spindle-slurm-srun.sh (NEW)
- PODMAN.md (MODIFIED - CPU pinning limitations documented)
Notes:
- CPU pinning attempted but not supported in rootless podman (disabled)
- Trap disabled in test script for debugging (cleanup manually)
- Build time: ~6 minutes for base, ~2 minutes for testing
- generate_config.sh must run before build to create slurmdbd.conf
Summary: Serial container works, Flux needs debugging, Slurm works perfectly!
Ready to debug multiple-commpaths issue.
Per Claude: ● Add Spindle Slurm rshlaunch container for podman (SSH blocked) Ports the Slurm rshlaunch test container to podman. Build succeeds and cluster starts, but SSH connections fail due to LC-specific setgroups issue in sshd's privilege separation. Status: ✓ Image builds successfully (reuses spindle-slurm-base) ✓ 7-container cluster starts (MariaDB + slurmdbd + slurmctld + 4 workers) ✓ Slurm daemons connect and allocations succeed ✗ SSH connections reset during key exchange ✗ Tests fail: "Connection reset by port 22" Root cause: SSH privilege separation setgroups failure sshd privilege separation calls setgroups() which fails on LC systems in rootless podman with "Invalid argument". This is the same setgroups issue we fixed for apt-get in hello-world-01. Debug output from `sshd -ddd`: debug3: privsep user:group 101:65534 [preauth] setgroups: Invalid argument [preauth] debug1: do_cleanup [preauth] Connection reset by port 22 Attempted fixes: 1. ✓ Started sshd in entrypoint (was missing) 2. ✓ Regenerated SSH host keys at runtime 3. ✓ Created /run/sshd directory 4. ✓ Fixed SSH key permissions in setup_ssh.sh 5. ✗ UsePrivilegeSeparation=no - still fails (may need sshd rebuild) Key differences from srun: - Uses --with-rsh-launch --with-rsh-cmd=/usr/bin/ssh in configure - Requires passwordless SSH between nodes via setup_ssh.sh - Starts sshd in entrypoint for inter-node communication Files: - containers/spindle-slurm-ubuntu/testing/Dockerfile.podman (NEW) - containers/spindle-slurm-ubuntu/testing/conf/slurm.conf.podman (NEW - copied from srun) - containers/spindle-slurm-ubuntu/testing/scripts/setup_slurm.sh.podman (NEW - copied from srun) - containers/spindle-slurm-ubuntu/testing/scripts/entrypoint.sh.podman (NEW - adds sshd startup) - scripts/podman/build-spindle-slurm-rshlaunch.sh (NEW) - scripts/podman/test-spindle-slurm-rshlaunch.sh (NEW) Possible solutions (not attempted): - Rebuild sshd without privilege separation support - Use a different SSH implementation (dropbear, etc.) - Patch sshd to skip setgroups call - Use rootful podman (requires sudo) Notes: - SSH works fine in Docker (doesn't hit LC setgroups restrictions) - rshlaunch is less common than srun for Spindle testing - Serial and srun containers fully work - sufficient for most testing
[save|load]-images allows the slurm image to be built once and copied to compute nodes. test-spindle-slurm-srun-parallel.sh allows multiple simultaneous runs without filesystem and network name conflicts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress.
Enables containerization similar to Docker+GitHub using rootless podman.