feat(helm): make gateway PVC size and StorageClass configurable (#3854)

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>

Author: lunarwhite

deploy/helm/openshell/README.md

@@ -203,6 +203,36 @@ server:
   dbUrl: "sqlite:/var/openshell/openshell.db"
 ```
 
+#### Data PVC storage
+
+Size the gateway data claim, `openshell-data`, and select its StorageClass with
+`server.dataStorageSize` and `server.dataStorageClass`. Omitting both
+requests `1Gi` from the cluster's default StorageClass. Set
+`server.dataStorageClass` on clusters with no default StorageClass,
+otherwise the gateway PVC stays Pending. Both values are ignored under
+`workload.kind=deployment`, which mounts no PVC.
+
+```yaml
+server:
+  dataStorageSize: 20Gi
+  dataStorageClass: fast-ssd
+```
+
+`volumeClaimTemplates` is immutable, so set both at install time. To grow an
+existing claim, expand it in place when its StorageClass allows expansion, then
+recreate the StatefulSet so its template matches:
+
+```shell
+kubectl -n openshell patch pvc openshell-data-openshell-0 \
+  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
+kubectl -n openshell delete statefulset openshell --cascade=orphan
+helm upgrade openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <installed-version> \
+  --reuse-values -n openshell --set server.dataStorageSize=20Gi
+```
+
+A recreated StatefulSet re-adopts existing claims unchanged, so changing
+StorageClass means copying the data to a new claim.
+
 #### External PostgreSQL
 
 Use external PostgreSQL when the gateway should connect to a database managed
@@ -482,6 +512,8 @@ discovery endpoint or its TLS CA.
 | server.credentialDrivers.vault.timeoutSecs | string | `""` | HTTP request timeout in seconds. Empty = driver default. |
 | server.credentialDrivers.vault.tokenPath | string | `""` | Mounted token file path when authMethod is token_file. |
 | server.credentialStorage.existingSecret | string | `""` | Name of a pre-existing Secret containing the key-encryption key. When set, the chart does NOT generate a new Secret; it references this one instead. The Secret must contain a key named "key-encryption-key" with a base64-encoded 32-byte value. Required for GitOps workflows that render manifests with `helm template` (where `lookup` is unavailable). |
+| server.dataStorageClass | string | `""` | Kubernetes StorageClass for the gateway data PVC (StatefulSet workload only). Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the gateway PVC stays Pending. Use a block-backed class, not NFS or other network filesystems. Immutable after install. |
+| server.dataStorageSize | string | `""` | Storage request for the gateway data PVC, `openshell-data` (StatefulSet workload only). Uses Kubernetes quantity syntax and requires an explicit unit ("20Gi", not "20"). Empty = built-in default (1Gi). Growing an existing release's claim is PVC expansion, not a values change. |
 | server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). SQLite runs in WAL mode and needs a local block-backed volume, not NFS or other network filesystems. |
 | server.defaultRuntimeClassName | string | `""` | Default Kubernetes runtimeClassName for sandbox pods. Applied when a CreateSandbox request does not specify one. Empty (default) = omit the field, using the cluster's default RuntimeClass. Set to a RuntimeClass name (e.g. "kata-containers", "nvidia") to apply it to all sandboxes that don't explicitly override it. |
 | server.disableTls | bool | `false` | Disable TLS entirely - the server listens on plaintext HTTP. Set to true when a reverse proxy / tunnel terminates TLS at the edge. |

deploy/helm/openshell/README.md.gotmpl

@@ -204,6 +204,36 @@ server:
   dbUrl: "sqlite:/var/openshell/openshell.db"
 ```
 
+#### Data PVC storage
+
+Size the gateway data claim, `openshell-data`, and select its StorageClass with
+`server.dataStorageSize` and `server.dataStorageClass`. Omitting both
+requests `1Gi` from the cluster's default StorageClass. Set
+`server.dataStorageClass` on clusters with no default StorageClass,
+otherwise the gateway PVC stays Pending. Both values are ignored under
+`workload.kind=deployment`, which mounts no PVC.
+
+```yaml
+server:
+  dataStorageSize: 20Gi
+  dataStorageClass: fast-ssd
+```
+
+`volumeClaimTemplates` is immutable, so set both at install time. To grow an
+existing claim, expand it in place when its StorageClass allows expansion, then
+recreate the StatefulSet so its template matches:
+
+```shell
+kubectl -n openshell patch pvc openshell-data-openshell-0 \
+  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
+kubectl -n openshell delete statefulset openshell --cascade=orphan
+helm upgrade openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <installed-version> \
+  --reuse-values -n openshell --set server.dataStorageSize=20Gi
+```
+
+A recreated StatefulSet re-adopts existing claims unchanged, so changing
+StorageClass means copying the data to a new claim.
+
 #### External PostgreSQL
 
 Use external PostgreSQL when the gateway should connect to a database managed

deploy/helm/openshell/templates/_helpers.tpl

@@ -624,4 +624,8 @@ Validate chart values that Helm would otherwise accept silently.
 {{- if kindIs "invalid" .Values.server.tls.clientCaSecretName -}}
 {{- fail "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" -}}
 {{- end -}}
+{{- $dataStorageSize := .Values.server.dataStorageSize -}}
+{{- if and (eq $workloadKind "statefulset") $dataStorageSize (not (regexMatch "^[0-9]+(\\.[0-9]+)?([KMGTPE]i|[kMGTPE])$" (toString $dataStorageSize))) -}}
+{{- fail "server.dataStorageSize must be a Kubernetes quantity with a unit, such as 20Gi" -}}
+{{- end -}}
 {{- end }}

deploy/helm/openshell/templates/statefulset.yaml

@@ -23,7 +23,10 @@ spec:
         name: openshell-data
       spec:
         accessModes: ["ReadWriteOnce"]
+        {{- with .Values.server.dataStorageClass }}
+        storageClassName: {{ . | quote }}
+        {{- end }}
         resources:
           requests:
-            storage: 1Gi
+            storage: {{ .Values.server.dataStorageSize | default "1Gi" | quote }}
 {{- end }}

deploy/helm/openshell/tests/gateway_persistence_test.yaml

@@ -0,0 +1,44 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite: gateway data persistence
+templates:
+  - templates/gateway-config.yaml
+  - templates/statefulset.yaml
+release:
+  name: openshell
+  namespace: my-namespace
+
+tests:
+  - it: requests 1Gi from the cluster default StorageClass by default
+    template: templates/statefulset.yaml
+    asserts:
+      - equal:
+          path: spec.volumeClaimTemplates[0].metadata.name
+          value: openshell-data
+      - equal:
+          path: spec.volumeClaimTemplates[0].spec.resources.requests.storage
+          value: 1Gi
+      - notExists:
+          path: spec.volumeClaimTemplates[0].spec.storageClassName
+
+  - it: renders an explicitly set data storage size and StorageClass
+    template: templates/statefulset.yaml
+    set:
+      server.dataStorageSize: 20Gi
+      server.dataStorageClass: fast-ssd
+    asserts:
+      - equal:
+          path: spec.volumeClaimTemplates[0].spec.resources.requests.storage
+          value: 20Gi
+      - equal:
+          path: spec.volumeClaimTemplates[0].spec.storageClassName
+          value: fast-ssd
+
+  - it: rejects a data storage size without a unit
+    template: templates/statefulset.yaml
+    set:
+      server.dataStorageSize: 20
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.dataStorageSize must be a Kubernetes quantity with a unit, such as 20Gi"

deploy/helm/openshell/values.yaml

@@ -363,6 +363,17 @@ server:
   # in WAL mode and needs a local block-backed volume, not NFS or other network
   # filesystems.
   dbUrl: "sqlite:/var/openshell/openshell.db"
+  # -- Storage request for the gateway data PVC, `openshell-data` (StatefulSet
+  # workload only). Uses Kubernetes quantity syntax and requires an explicit unit
+  # ("20Gi", not "20"). Empty = built-in default (1Gi). Growing an existing
+  # release's claim is PVC expansion, not a values change.
+  dataStorageSize: ""
+  # -- Kubernetes StorageClass for the gateway data PVC (StatefulSet workload
+  # only). Empty (default) = omit storageClassName, using the cluster's default
+  # StorageClass. Set this on clusters with no default StorageClass, otherwise
+  # the gateway PVC stays Pending. Use a block-backed class, not NFS or other
+  # network filesystems. Immutable after install.
+  dataStorageClass: ""
   # -- Name of a pre-existing Opaque Secret containing a PostgreSQL
   # connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL
   # from this Secret instead of using dbUrl. The Secret must contain a

docs/kubernetes/setup.mdx

@@ -294,6 +294,8 @@ The most commonly changed values are:
 | `server.sandboxNamespace` | Namespace where sandbox pods are created. Defaults to the Helm release namespace when left empty. |
 | `workspaceResources.enabled` | Create namespace-scoped sandbox prerequisites from the gateway chart. Disable when installing the workspace chart separately. |
 | `server.externalDbSecret` | Secret containing a PostgreSQL connection URI in the `uri` key. Use when the database is managed outside the chart. |
+| `server.dataStorageSize` | Storage request for the gateway data PVC under `workload.kind=statefulset`. Defaults to `1Gi`. Requires an explicit unit, such as `20Gi`. |
+| `server.dataStorageClass` | `StorageClass` for the gateway data PVC under `workload.kind=statefulset`. Empty (default) omits `storageClassName` and uses the cluster's default `StorageClass`. Use a block-backed class, not NFS. |
 | `server.telemetryEnabled` | Enable anonymous OpenShell telemetry from the gateway and its sandbox supervisors. Set to `false` to opt out. |
 | `sandbox.image.repository` / `sandbox.image.tag` / `sandbox.image.digest` | Default sandbox image used when a sandbox does not specify one. |
 | `sandboxRuntime.image.registry` / `sandboxRuntime.image.repository` / `sandboxRuntime.image.tag` / `sandboxRuntime.image.digest` | Trusted workload-side image that provides the `openshell-sandbox` binary. A digest takes precedence over the tag. |
@@ -317,6 +319,10 @@ helm upgrade --install openshell \
   --values my-values.yaml
 ```
 
+<Warning>
+`server.dataStorageSize` and `server.dataStorageClass` feed the gateway StatefulSet's immutable `volumeClaimTemplates`. Set them at install time. Changing them later fails the upgrade and does not move existing data; refer to the [Helm chart README](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md#data-pvc-storage) for migration steps.
+</Warning>
+
 To use private sandbox images, create a `kubernetes.io/dockerconfigjson` Secret
 in the sandbox namespace and reference its name:
 

skills/debug-openshell-cluster/SKILL.md

@@ -1176,7 +1176,8 @@ credential failures.
 | Docker or Podman sandbox never registers | Wrong gateway endpoint, unavailable host networking, or supervisor startup failure | Gateway logs and supervisor container logs |
 | CPU-only VM startup reports that read-only `/proc` cannot be removed | Older supervisors can infer GPU requirements from host devices | Check gateway, VM driver, and bundled supervisor versions; use matching updated artifacts. CPU-only VMs do not require read-write `/proc`. |
 | Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
-| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
+| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>`; if the gateway PVC is unbound because the cluster has no default `StorageClass`, set `server.dataStorageClass` to a valid one, and size the claim with `server.dataStorageSize` |
+| `helm upgrade` fails on a forbidden update to the gateway StatefulSet after changing `server.dataStorageSize` or `server.dataStorageClass` | `volumeClaimTemplates` is immutable on an existing StatefulSet | `kubectl -n openshell delete statefulset openshell --cascade=orphan`, then re-upgrade; the orphaned PVC keeps its original size and class, so grow it with in-place PVC expansion or copy the data to a new claim |
 | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
 | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
 | Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full lock pool on one replica (4 connections unless `database_lock_max_connections` is set), a slow PostgreSQL, or older gateways still running during an upgrade | `openshell_server_mutation_lock_timeouts_total`, `openshell_server_mutation_lock_connections_in_use` against its capacity, the `detail` field of `mutation lock acquisition timed out` in gateway logs, `mutation lock held longer than the lock wait timeout` warnings and `openshell_server_mutation_lock_hold_seconds`, the `pg_locks` query in Step 6 |