feat(helm): configure gateway OCSF JSONL output (#3876)

Previously, Helm installations could not enable the gateway OCSF JSONL
destination through chart values because generated `gateway.toml` omitted the
`openshell.gateway.ocsf_log` table.

Now, setting `server.ocsfLog.enabled` renders the path, optional schema
version, rotation, retention, and queue limits into gateway configuration.
Output is disabled by default. The default path, `/tmp/gateway-ocsf.jsonl`,
is writable in the gateway container with either the StatefulSet or
Deployment workload, so enabling output does not require persistent storage.
Invalid schema versions, rotation values, non-positive limits, or an empty
path while enabled fail chart rendering.

Additionally, `server.extraVolumes` and `server.extraVolumeMounts` add
operator-supplied volumes to the gateway pod, so operators who want records
to survive restarts can place the OCSF path on persistent storage without
replacing chart-generated configuration.

The gateway pod's default termination grace period rises from 5 to 30
seconds. Gateway shutdown can spend up to 10 seconds on supervisor session
cleanup before allowing 5 seconds to drain queued OCSF records, so the
5-second default risked a SIGKILL before the final records were written.
The grace period is only an upper bound: the gateway exits as soon as its
shutdown completes.

Refs #2762

Signed-off-by: Kris Hicks <khicks@nvidia.com>

Author: krishicks

deploy/helm/openshell/README.md

@@ -355,7 +355,7 @@ discovery endpoint or its TLS CA.
 | pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. |
 | podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. |
 | podLabels | object | `{}` | Extra labels to add to the gateway pod. |
-| podLifecycle.terminationGracePeriodSeconds | int | `5` | Grace period, in seconds, before Kubernetes terminates the gateway pod. |
+| podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. |
 | podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. |
 | probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. |
 | probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. |
@@ -420,12 +420,21 @@ discovery endpoint or its TLS CA.
 | server.enableUserNamespaces | bool | `false` | Enable Kubernetes user namespace isolation (hostUsers: false) for sandbox pods. Requires Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), plus a supporting container runtime and Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced. |
 | server.enableWebsocketTunnel | bool | `false` | Enable the WebSocket tunnel used by CLI/SDK clients behind an authenticated edge proxy. Leave disabled for direct gateway installs. |
 | server.externalDbSecret | string | `""` | Name of a pre-existing Opaque Secret containing a PostgreSQL connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL from this Secret instead of using dbUrl. The Secret must contain a `uri` key, e.g. postgresql://user:pass@host:5432/dbname. |
+| server.extraVolumeMounts | list | `[]` | Additional volume mounts for the gateway container. |
+| server.extraVolumes | list | `[]` | Additional volumes for the gateway pod. |
 | server.grpcEndpoint | string | `""` | gRPC endpoint sandboxes call back into the gateway. Leave empty to derive it from the chart fullname, release namespace, service port, and disableTls flag, for example https://openshell.openshell.svc.cluster.local:8080. Override only when sandboxes must reach the gateway via a different hostname (e.g. an external ingress or a host alias). |
 | server.grpcRateLimit.requests | int | `0` | Maximum gRPC requests allowed per window. Must be positive (alongside windowSeconds) to enable rate limiting; 0 (default) disables it. |
 | server.grpcRateLimit.windowSeconds | int | `0` | gRPC rate-limit window length in seconds. Must be positive (alongside requests) to enable rate limiting; 0 (default) disables it. |
 | server.hostGatewayIP | string | `""` | Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get hostAliases entries mapping host.docker.internal and host.openshell.internal to this IP, allowing them to reach services running on the Docker host. Auto-detected by the cluster entrypoint script. |
 | server.logLevel | string | `"info"` | Gateway log level. |
 | server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. |
+| server.ocsfLog.enabled | bool | `false` | Write gateway OCSF events as JSONL. |
+| server.ocsfLog.maxFiles | int | `7` | Rotated files retained when rotation is daily. |
+| server.ocsfLog.path | string | `"/tmp/gateway-ocsf.jsonl"` | OCSF JSONL path. The default is writable in the gateway container but does not persist across restarts. To keep records, mount a volume with server.extraVolumes and server.extraVolumeMounts and set a path on it. When replicas share the volume, set subPathExpr: $(OPENSHELL_POD_NAME) on the mount so each replica writes its own file. |
+| server.ocsfLog.queueCapacity | int | `10000` | Maximum records waiting for the file writer. |
+| server.ocsfLog.queueMaxBytes | int | `16777216` | Maximum encoded bytes waiting for the file writer. |
+| server.ocsfLog.rotation | string | `"daily"` | Rotate the active file daily in UTC, or never. |
+| server.ocsfLog.schemaVersion | string | `""` | Optional OCSF downgrade target. Empty emits native OCSF 1.8.0. Supported values: "1.1", "1.3". |
 | server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. |
 | server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. |
 | server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |

deploy/helm/openshell/templates/_gateway-workload.tpl

@@ -184,6 +184,9 @@ spec:
           mountPath: {{ dir .Values.server.providerTokenGrants.spiffe.workloadApiSocketPath | quote }}
           readOnly: true
         {{- end }}
+        {{- with .Values.server.extraVolumeMounts }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
       ports:
         - name: grpc
           containerPort: {{ .Values.service.port }}
@@ -291,6 +294,9 @@ spec:
         driver: csi.spiffe.io
         readOnly: true
     {{- end }}
+    {{- with .Values.server.extraVolumes }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
   {{- with .Values.nodeSelector }}
   nodeSelector:
     {{- toYaml . | nindent 4 }}

deploy/helm/openshell/templates/gateway-config.yaml

@@ -14,6 +14,7 @@ One value is intentionally NOT rendered here:
 */}}
 {{- $credentialDrivers := list -}}
 {{- $otlp := .Values.server.otlp | default dict -}}
+{{- $ocsfLog := .Values.server.ocsfLog | default dict -}}
 {{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled -}}
 {{- $credentialDrivers = append $credentialDrivers "kubernetes-secrets" -}}
 {{- end -}}
@@ -88,6 +89,41 @@ data:
     {{- end }}
     {{- end }}
 
+    {{- if $ocsfLog.enabled }}
+    {{- if not $ocsfLog.path -}}
+    {{- fail "server.ocsfLog.path must be set when server.ocsfLog.enabled is true" -}}
+    {{- end }}
+
+    {{- $ocsfRotation := $ocsfLog.rotation | default "daily" -}}
+    {{- if not (has $ocsfRotation (list "daily" "never")) -}}
+    {{- fail "server.ocsfLog.rotation must be daily or never" -}}
+    {{- end }}
+    {{- $ocsfQueueCapacity := int (ternary 10000 $ocsfLog.queueCapacity (kindIs "invalid" $ocsfLog.queueCapacity)) -}}
+    {{- $ocsfQueueMaxBytes := int (ternary 16777216 $ocsfLog.queueMaxBytes (kindIs "invalid" $ocsfLog.queueMaxBytes)) -}}
+    {{- if or (lt $ocsfQueueCapacity 1) (lt $ocsfQueueMaxBytes 1) -}}
+    {{- fail "server.ocsfLog.queueCapacity and queueMaxBytes must be positive" -}}
+    {{- end }}
+    [openshell.gateway.ocsf_log]
+    path = {{ $ocsfLog.path | quote }}
+    {{- $ocsfSchemaVersion := $ocsfLog.schemaVersion | default "" -}}
+    {{- if not (has $ocsfSchemaVersion (list "" "1.1" "1.3")) -}}
+    {{- fail "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3" -}}
+    {{- end }}
+    {{- if $ocsfSchemaVersion }}
+    schema_version = {{ $ocsfSchemaVersion | quote }}
+    {{- end }}
+    rotation = {{ $ocsfRotation | quote }}
+    {{- if eq $ocsfRotation "daily" }}
+    {{- $ocsfMaxFiles := int (ternary 7 $ocsfLog.maxFiles (kindIs "invalid" $ocsfLog.maxFiles)) -}}
+    {{- if lt $ocsfMaxFiles 1 -}}
+    {{- fail "server.ocsfLog.maxFiles must be positive when rotation is daily" -}}
+    {{- end }}
+    max_files = {{ $ocsfMaxFiles }}
+    {{- end }}
+    queue_capacity = {{ $ocsfQueueCapacity }}
+    queue_max_bytes = {{ $ocsfQueueMaxBytes }}
+    {{- end }}
+
     {{- if not .Values.server.disableTls }}
 
     [openshell.gateway.tls]

deploy/helm/openshell/tests/gateway_config_test.yaml

@@ -303,6 +303,166 @@ tests:
           path: data["gateway.toml"]
           pattern: '\[openshell\.gateway\.otlp\]'
 
+  - it: omits OCSF JSONL output by default
+    template: templates/gateway-config.yaml
+    asserts:
+      - notMatchRegex:
+          path: data["gateway.toml"]
+          pattern: '\[openshell\.gateway\.ocsf_log\]'
+
+  - it: treats a null OCSF log map as disabled
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog: null
+    asserts:
+      - notMatchRegex:
+          path: data["gateway.toml"]
+          pattern: '\[openshell\.gateway\.ocsf_log\]'
+
+  - it: omits OCSF JSONL output when a path is set without enabling it
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
+    asserts:
+      - notMatchRegex:
+          path: data["gateway.toml"]
+          pattern: '\[openshell\.gateway\.ocsf_log\]'
+
+  - it: writes OCSF JSONL to the default container path when enabled
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog.enabled: true
+    asserts:
+      - matchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/tmp/gateway-ocsf\.jsonl"'
+
+  - it: rejects enabled OCSF JSONL output without a path
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: ""
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.path must be set when server.ocsfLog.enabled is true"
+
+  - it: renders OCSF JSONL output when configured
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
+      server.ocsfLog.schemaVersion: "1.3"
+      server.ocsfLog.rotation: daily
+      server.ocsfLog.maxFiles: 14
+      server.ocsfLog.queueCapacity: 20000
+      server.ocsfLog.queueMaxBytes: 33554432
+    asserts:
+      - matchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?path\s*=\s*"/var/openshell/gateway-ocsf\.jsonl".*?schema_version\s*=\s*"1\.3".*?rotation\s*=\s*"daily".*?max_files\s*=\s*14.*?queue_capacity\s*=\s*20000.*?queue_max_bytes\s*=\s*33554432'
+
+  - it: omits OCSF retention when rotation is disabled
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
+      server.ocsfLog.rotation: never
+    asserts:
+      - matchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?rotation\s*=\s*"never"'
+      - notMatchRegex:
+          path: data["gateway.toml"]
+          pattern: 'max_files\s*='
+
+  - it: rejects an unsupported OCSF rotation mode
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
+      server.ocsfLog.rotation: hourly
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.rotation must be daily or never"
+
+  - it: rejects a zero OCSF queueCapacity
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.queueCapacity: 0
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.queueCapacity and queueMaxBytes must be positive"
+
+  - it: rejects a zero OCSF queueMaxBytes
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.queueMaxBytes: 0
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.queueCapacity and queueMaxBytes must be positive"
+
+  - it: rejects a zero OCSF maxFiles
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.maxFiles: 0
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.maxFiles must be positive when rotation is daily"
+
+  - it: uses default OCSF limits when they are null
+    template: templates/gateway-config.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.queueCapacity: null
+      server.ocsfLog.queueMaxBytes: null
+      server.ocsfLog.maxFiles: null
+    asserts:
+      - matchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?ms)\[openshell\.gateway\.ocsf_log\].*?max_files\s*=\s*7.*?queue_capacity\s*=\s*10000.*?queue_max_bytes\s*=\s*16777216'
+
+  - it: rejects an unsupported OCSF schema version
+    template: templates/statefulset.yaml
+    set:
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: /var/openshell/gateway-ocsf.jsonl
+      server.ocsfLog.schemaVersion: "1.8"
+    asserts:
+      - failedTemplate:
+          errorMessage: "server.ocsfLog.schemaVersion must be empty, 1.1, or 1.3"
+
+  - it: mounts extra gateway volumes for OCSF output in a Deployment
+    template: templates/deployment.yaml
+    set:
+      workload.kind: deployment
+      server.externalDbSecret: my-pg-secret
+      server.ocsfLog.enabled: true
+      server.ocsfLog.path: /var/log/openshell/gateway-ocsf.jsonl
+      server.extraVolumes:
+        - name: ocsf-log
+          persistentVolumeClaim:
+            claimName: openshell-ocsf-log
+      server.extraVolumeMounts:
+        - name: ocsf-log
+          mountPath: /var/log/openshell
+          subPathExpr: $(OPENSHELL_POD_NAME)
+    asserts:
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: ocsf-log
+            persistentVolumeClaim:
+              claimName: openshell-ocsf-log
+      - contains:
+          path: spec.template.spec.containers[0].volumeMounts
+          content:
+            name: ocsf-log
+            mountPath: /var/log/openshell
+            subPathExpr: $(OPENSHELL_POD_NAME)
+
   - it: mounts the OIDC CA bundle when TLS is disabled
     template: templates/statefulset.yaml
     set:
@@ -1242,3 +1402,19 @@ tests:
       - matchRegex:
           path: data["gateway.toml"]
           pattern: "(?ms)\\[openshell\\.drivers\\.kubernetes\\].*?image_pull_policy\\s*=\\s*\\\"never\\\".*?supervisor_image_pull_policy\\s*=\\s*\\\"never\\\""
+
+  - it: gives the gateway time to finish shutdown by default
+    template: templates/statefulset.yaml
+    asserts:
+      - equal:
+          path: spec.template.spec.terminationGracePeriodSeconds
+          value: 30
+
+  - it: keeps an explicit termination grace period
+    template: templates/statefulset.yaml
+    set:
+      podLifecycle.terminationGracePeriodSeconds: 3
+    asserts:
+      - equal:
+          path: spec.template.spec.terminationGracePeriodSeconds
+          value: 3

deploy/helm/openshell/values.yaml

@@ -212,8 +212,10 @@ agentSandbox:
 
 # Pod restart behavior and health probe tuning.
 podLifecycle:
-  # -- Grace period, in seconds, before Kubernetes terminates the gateway pod.
-  terminationGracePeriodSeconds: 5
+  # -- Maximum time, in seconds, Kubernetes waits for the gateway to exit before
+  # killing it. The gateway exits as soon as shutdown completes; the limit
+  # covers supervisor session cleanup and draining queued OCSF records.
+  terminationGracePeriodSeconds: 30
 
 probes:
   startup:
@@ -268,6 +270,31 @@ server:
     endpoint: ""
     # -- Gateway OpenTelemetry service name. Empty uses openshell-gateway.
     serviceName: ""
+  # Gateway-native OCSF JSONL output.
+  ocsfLog:
+    # -- Write gateway OCSF events as JSONL.
+    enabled: false
+    # -- OCSF JSONL path. The default is writable in the gateway container but
+    # does not persist across restarts. To keep records, mount a volume with
+    # server.extraVolumes and server.extraVolumeMounts and set a path on it.
+    # When replicas share the volume, set subPathExpr: $(OPENSHELL_POD_NAME)
+    # on the mount so each replica writes its own file.
+    path: /tmp/gateway-ocsf.jsonl
+    # -- Optional OCSF downgrade target. Empty emits native OCSF 1.8.0.
+    # Supported values: "1.1", "1.3".
+    schemaVersion: ""
+    # -- Rotate the active file daily in UTC, or never.
+    rotation: daily
+    # -- Rotated files retained when rotation is daily.
+    maxFiles: 7
+    # -- Maximum records waiting for the file writer.
+    queueCapacity: 10000
+    # -- Maximum encoded bytes waiting for the file writer.
+    queueMaxBytes: 16777216
+  # -- Additional volumes for the gateway pod.
+  extraVolumes: []
+  # -- Additional volume mounts for the gateway container.
+  extraVolumeMounts: []
   # -- Enable anonymous OpenShell telemetry from the gateway and the sandbox
   # supervisors it launches.
   telemetryEnabled: true

docs/how-it-works/gateways/configuration.mdx

@@ -303,6 +303,8 @@ queue_max_bytes = 16777216
 
 Unknown fields are rejected. This is one file destination, not an exporter registry. The existing `ocsf_json_enabled` sandbox setting remains independent.
 
+For Helm installations, set `server.ocsfLog.enabled` to render this table and optionally set `server.ocsfLog.schemaVersion`. The default `server.ocsfLog.path`, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload but does not persist across pod restarts. To keep records, mount a volume with `server.extraVolumes` and `server.extraVolumeMounts` and set the path on it. When replicas share a volume, add `subPathExpr: $(OPENSHELL_POD_NAME)` to the mount so each replica writes its own file.
+
 Give each gateway replica its own writable file and an external shipper read access to the containing directory. New files use owner-only permissions on Unix; arrange shipper access deliberately. Rotation renames the active file to a date- and UUID-suffixed sibling. The shipper must follow rotated files and checkpoint its progress. Do not use multiple writers or external copy-truncate rotation on this path.
 
 The gateway queues OCSF independently of `RUST_LOG`, excludes ordinary diagnostics, and preserves native event IDs. It writes up to 100 records per batch with a 500 ms batching interval. One additional bounded batch can be in flight. File errors do not stop sandbox execution: failed writes are not replayed, and subsequent records are discarded during reopen backoff. A restarted writer removes an incomplete trailing line before appending.