IOOS Cloud Sandbox

The IOOS Cloud Sandbox is a collaborative platform for running regional coastal models in the cloud.

It provides repeatable configurations, model code and required libraries, input data, and analysis of model outputs. The Sandbox provisions Cloud HPC to aid in the development of services and models, and also supports running and validating models. The Sandbox is intended for use across industries and is inclusive to anyone who wants to develop, enhance, and run coastal models.

Use cases

• Inter-organization collaboration
• University graduate courses
• Hindcasts/Reanalysis
• Research to Operations (R2O)
• On-demand HPC capacity
• Quasi-operational HPC fail-over (natural disasters, data-center problems)
• AI/ML training

Deployment options

• Self deployed/hosted: Deploy using your own AWS cloud account
• IOOS hosted: NOAA AWS Government Cloud – Lynker contract
• RPS hosted: Isolated secure Virtual Private Cloud (VPC)

The Cloud Sandbox has been used to run:

• Operational versions of the NOSOFS ROMS and FVCOM Models
• LiveOcean model from the University of Washington
• WRF/ROMS ESMF Coupled (Hurricane Irene test case)
• WRF/ROMS/SWAN
• ADCIRC (Hurricane Florence test case)
• CORA ADCIRC model
• SCHISM
• National Water Model
• UCLA-ROMS
• UFS-Coastal
• ECCOFS
• SECOFS
• NECOFS

Documentation

Comprehensive guides and instructions for provisioning, configuring, and executing workflows on the Cloud Sandbox:

• Infrastructure Deployment Guide: Instructions for deploying and hosting a Cloud Sandbox instance on AWS using Terraform.
• Cloudflow Workflow System Overview: An architectural overview of the Cloudflow orchestration framework and Prefect server setup.
• Prefect v3 Setup & Troubleshooting Guide: Configuration details for running the local Prefect v3 server in the background, persisting results, establishing SSH tunnels to the dashboard UI (http://localhost:4200/dashboard), and managing active flow runs via CLI.
• Local Python Miniforge3 Installation Guide: Step-by-step instructions for installing a isolated Miniforge3 package manager and compiling a Cloudflow-compatible Python environment under your personal directory on the EFS volume of a given head node.
• Zombie Job Checklist: Operational protocols, environment flag standards (I_MPI), Python deadlock hooks (mpi4py), Dask fast-fail configurations, and health-check steps to detect, prevent, and terminate hung or silent "zombie" processes on AWS EC2 instances.

I want to...

• Run a model in an existing cloud sandbox

  • NOS OFS Models
  • LiveOcean Model
  • CORA
  • UCLA-ROMS
  • UFS-Coastal
  • NECOFS
  • SECOFS
  • ECCOFS
  • PYTHON
  • Basic Model Implementation

• Integrate a new model into the cloud sandbox

• Deploy a new cloud sandbox

Software Stack

The Cloud Sandbox uses CloudFlow to orchestrate the execution of the model. CloudFlow is a customized workflow for running models built on Prefect Workflows.

Software Architecture

The Cloud Sandbox uses Terraform to deploy resources to the cloud. The sandbox currently only supports the Amazon Web Services (AWS) cloud platform.

• Head Node: The head node is the machine that runs the CloudFlow scheduler. Users can SSH into this machine to run CloudFlow tasks.
• Preconfigured AMI: This Amazon Machine Image (AMI) contains all of the necessary code to run the model. This image will run on the worker nodes.
• Worker Nodes: The worker nodes are provisioned by Terraform and run CloudFlow tasks. This runs the preconfigured AMI.