Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,16 @@ usemenu:
url: /guide-e4s-bot/
- title: "E4S Quick Start"
url: /quick-start/
- title: "PyTorch Quick Start"
url: /quick-start-pytorch/
- title: "TensorFlow Quick Start"
url: /quick-start-tensorflow/
- title: "NeMo Quick Start"
url: /quick-start-nemo/
- title: "BioNeMo Quick Start"
url: /quick-start-bionemo/
- title: "vLLM Quick Start"
url: /quick-start-vllm/
- title: "HYPRE Quick Start"
url: /quick-start-hypre/
- title: "LAMMPS Quick Start"
Expand All @@ -41,6 +51,8 @@ usemenu:
url: /container-installation/
- title: "E4S Container Launch"
url: /container-launch/
- title: "Frontier E4S Container Demo"
url: /frontier-container-demo/
- title: "Test with E4S"
url: /test-with-e4s/
- title: "ParaTools Pro for E4S™"
Expand Down
113 changes: 113 additions & 0 deletions frontier-container-demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
layout: single
title: "Frontier E4S Container Demo"
permalink: /frontier-container-demo/
classes: wide
sidebar:
nav: "usemenu"
---

{% include e4s-page-actions.html %}

# Frontier E4S container demo

This example demonstrates a realistic E4S workflow on OLCF Frontier using the ROCm-enabled E4S container image. The goal is to show how to:

- fetch a prebuilt E4S Apptainer/Singularity image into a scratch area,
- build and run a small example inside the container using Spack provided software,
- allocate a GPU job on Frontier,
- and execute a real GPU application from the container environment.

This is a good fit for users who want to prototype GPU workloads on Frontier without maintaining a separate host build environment for every dependency.

## Prerequisites

Before starting, make sure you are logged into Frontier with a valid project allocation and that the standard OLCF environment is active. A typical workflow uses the container and MPI modules that are already provided by the system.

The commands below assume you have a project allocation and the ability to write to a scratch area under `$MEMBERWORK`.

## 1) Download the E4S ROCm image to scratch

Use a scratch directory under `$MEMBERWORK` so the image is stored on a location appropriate for large temporary data.

```bash
mkdir -p ${MEMBERWORK}/e4s-demo
cd ${MEMBERWORK}/e4s-demo

wget https://oaciss.nic.uoregon.edu/e4s/images/26.06/e4s-rocm90a-x86_64-26.06.sif
ls -lh e4s-rocm90a-x86_64-26.06.sif
```

This image is approximately 24 GB and may take around 20 minutes to download, depending on network conditions.

> There is also a CPU-only E4S image for MPI/thread-oriented workloads that do not rely on GPU-accelerated software. This ROCm image is the right choice for the GPU benchmark shown here. Other ROCm images are available for different GPU hardware versions.

## 2) Build a PETSc example inside the container

This step uses the container to source Spack, load a package, and compile a small PETSc example.

```bash
cd ${MEMBERWORK}/e4s-demo

apptainer exec --rocm e4s-rocm90a-x86_64-26.06.sif bash -c '
source /spack/share/spack/setup-env.sh 2>/dev/null || true
spack load petsc
PETSC_DIR=$(spack location -i petsc)
cp ${PETSC_DIR}/share/petsc/examples/src/snes/tutorials/ex19.c .
cp ${PETSC_DIR}/share/petsc/examples/src/snes/tutorials/makefile .
make ex19
'
```

This compiles a representative example from PETSc into the working directory inside the container. Critically, the build is executed within the container environment where the correct Spack packages and toolchain are already set up.

## 3) Request an allocation on Frontier

Use `salloc` to reserve compute resources before running the actual GPU workload.

```bash
salloc -A <project_id> -N 1 -t 60 -q batch
```

Once the allocation is active, move back into the demo directory and load the container support modules needed for GPU-aware containerized execution.

```bash
cd ${MEMBERWORK}/e4s-demo
module load apptainer-enable-gpu apptainer-enable-mpi
```

## 4) Run the workload in the container

The final step uses a single node with 8 tasks and one GPU per task. This matches the Frontier GPU setup and launches the benchmark inside the E4S container with the correct MPI and ROCm configuration.

```bash
srun -N 1 -n 8 --ntasks-per-node=8 --gpus-per-task=1 --gpu-bind=closest --mpi=pmi2 \
apptainer exec --rocm e4s-rocm90a-x86_64-26.06.sif \
bash -c "source /spack/share/spack/setup-env.sh && spack load slate && unset CUDA_VISIBLE_DEVICES HIP_VISIBLE_DEVICES; slate_tester --origin d --target d --dim 10240 --nb 512 gemm"
```

This command is intentionally simple to illustrate the overall pattern:

- the image is launched with ROCm support enabled,
- the Spack environment is loaded inside the container,
- and the GPU benchmark runs with the container-resident BLAS/LAPACK stack and host MPI/GPU scheduling.

## Why this workflow matters

This demonstration shows an E4S usage pattern that is common on HPC systems:

- container images provide a known-good scientific software stack,
- Spack within the container resolves dependencies consistently,
- and the host scheduler still controls the job placement and GPU assignments.

That combination makes it easier to reproduce performance studies and to run complex GPU software stacks without rebuilding a full environment on the host.

## Related E4S container pages

For additional container guidance and broader installation patterns, see:

- [E4S Container Installation](/container-installation/)
- [E4S Container Launch](/container-launch/)
- [Test with E4S](/test-with-e4s/)

Please contact the E4S team if you want help adapting this workflow to a different machine, MPI stack, or benchmark.
86 changes: 86 additions & 0 deletions use-bionemo-quick-start.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
layout: single
title: "BioNeMo Quick Start"
permalink: /quick-start-bionemo/
classes: wide
sidebar:
nav: "usemenu"
---

{% include e4s-page-actions.html %}

This page shows a BioNeMo workflow in the E4S 26.06 CUDA container. The examples here are intended to be run inside the Docker image shown below.

```bash
docker run --gpus all -it --rm \
--entrypoint bash \
-v "$PWD:/work" \
-w /work \
ecpe4s/e4s-cuda:26.06
```

## 1. Verify BioNeMo

The E4S container includes `bionemo-core` and related BioNeMo packages.

```bash
python3 - <<'PY'
import bionemo.core
import bionemo.moco
import bionemo.noodles
import bionemo.scdl

print('bionemo.core OK:', bionemo.core.__file__)
print('BioNeMo package imports OK')
PY
```

## 2. Run an ESM-2 inference example

This example loads an ESM-2 checkpoint and computes a sequence embedding with the container's GPU-enabled deep-learning stack.

Create a script:

```bash
cat > bionemo_esm2_recipe_demo.py <<'PY'
from transformers import AutoModel, AutoTokenizer
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = AutoModel.from_pretrained(
'nvidia/esm2_t6_8M_UR50D',
trust_remote_code=True,
).to(device)
tokenizer = AutoTokenizer.from_pretrained(
'nvidia/esm2_t6_8M_UR50D',
trust_remote_code=True,
)

sequence = 'MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTL'
inputs = tokenizer(sequence, return_tensors='pt').to(device)
with torch.no_grad():
output = model(**inputs)

print('Output embedding shape:', output.last_hidden_state.shape)
PY
```

Run the example:

```bash
python3 bionemo_esm2_recipe_demo.py
```

The first run downloads the checkpoint and may print model-loading details. The `UNEXPECTED` entries for `lm_head` weights can be ignored for this embedding model. A successful run ends with output similar to:

```text
Output embedding shape: torch.Size([1, 62, 320])
```

The exact sequence length can vary with the input sequence; the final dimension is the model's embedding size.

## 3. Typical next steps

After the container is running, you can adapt the example to your own protein sequences and use a mounted working directory for scripts and outputs.

For additional container guidance, see [E4S Container Installation](/container-installation/) and [E4S Container Launch](/container-launch/).
98 changes: 98 additions & 0 deletions use-nemo-quick-start.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
layout: single
title: "NeMo Quick Start"
permalink: /quick-start-nemo/
classes: wide
sidebar:
nav: "usemenu"
---

{% include e4s-page-actions.html %}

This page shows a NeMo workflow in the E4S 26.06 CUDA container. The examples here are intended to be run inside the Docker image shown below.

```bash
docker run --gpus all -it --rm \
--entrypoint bash \
-v "$PWD:/work" \
-w /work \
ecpe4s/e4s-cuda:26.06
```

## 1. Verify the NeMo packages

The container includes the NeMo packages used by this workflow: `megatron.core`, `megatron.bridge`, and `nemo_run`.

A quick import check:

```bash
python3 - <<'PY'
import megatron.core
import megatron.bridge
import nemo_run
print('megatron.core import OK')
print('megatron.bridge import OK')
print('nemo_run import OK:', getattr(nemo_run, '__version__', 'OK'))
PY
```

This confirms the NeMo stack is available for use inside the container.

## 2. Minimal Megatron-Bridge smoke test

The core pattern in this workflow is converting a Hugging Face model to Megatron's native format through `AutoBridge`.

Create a script:

```bash
cat > megatron_bridge_demo.py <<'PY'
import torch
from megatron.bridge import AutoBridge

print('Loading a small HF model and converting it to Megatron format...')
bridge = AutoBridge.from_hf_pretrained('Qwen/Qwen3-0.6B')
model_provider = bridge.to_megatron_provider(load_weights=True)

print('Bridge + Megatron provider constructed successfully.')
print(type(model_provider))
PY
```

Run it with the GPU selected:

```bash
python3 megatron_bridge_demo.py
```

This loads a small model and demonstrates the bridge and model-conversion workflow.

## 3. Run a local NeMo-Run experiment

Create a small task and launch it locally with `nemo_run`:

```bash
cat > nemo_run_demo.py <<'PY'
import nemo_run as run

task = run.Script(
inline="echo 'NeMo-Run experiment completed successfully.'"
)

with run.Experiment('e4s-quick-start') as experiment:
experiment.add(task, executor=run.LocalExecutor(), name='hello')
experiment.run(detach=False)
PY
python3 nemo_run_demo.py
```

The command should print `NeMo-Run experiment completed successfully.`.

## 4. Typical next steps

After the container is running, you can:

- test model conversion and bridge workflows,
- use `nemo_run` to launch experiments,
- and adapt the same pattern to your own training or evaluation scripts.

For additional container guidance, see [E4S Container Installation](/container-installation/) and [E4S Container Launch](/container-launch/).
Loading