fix(snap): migrate insecure gateway configs on refresh

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Author: drew

architecture/build.md

@@ -336,12 +336,17 @@ artifact and verifies the release run instead of listing every repository
 artifact.
 
 Snapd runs the gateway as a root-owned system service. Its generated client
-certificates reside in root-owned snap state and are unavailable to ordinary CLI
-users, so the Snap uses plaintext loopback transport and enables unauthenticated
-local users by default. Debian and RPM packages instead run systemd user services
-and use user-owned mTLS material. Bootstrap creates the default configuration
-only when it is missing. Sandbox-to-gateway sessions remain authenticated with
-gateway-minted JWTs.
+certificates reside in root-owned snap state. The installer copies the client
+bundle into the target user's private Snap state and registers the TLS endpoint;
+direct Snap installs require the same enrollment. The install and post-refresh
+hooks replace configs that explicitly enable plaintext or unauthenticated access
+with the secure Docker default, preserving one private backup. Snap refreshes
+restart the gateway so the migrated config takes effect immediately.
+
+Debian and RPM packages instead run systemd user services with user-owned mTLS
+material. Sandbox-to-gateway sessions remain authenticated with gateway-minted
+JWTs.
+
 The Debian qualification profile keeps candidate-image overrides outside the
 operator-owned gateway configuration: it writes a harness-owned file under
 `/var/lib/openshell-qualification` and selects it through the packaged systemd

docs/about/installation.mdx

@@ -112,18 +112,28 @@ sudo snap install openshell
 
 The snap does not migrate existing Debian, RPM, or Homebrew installs. Remove any existing installation first, then rerun the script with `OPENSHELL_ACK_BREAKING_UPGRADE=1`.
 
-The gateway runs as a system service at `http://127.0.0.1:17670` and reads `/var/snap/openshell/common/gateway.toml`.
-
-<Warning>
-The snap gateway allows unauthenticated access from the local host. Any local user or process can operate it. Do not expose it beyond the local host.
-</Warning>
-
-Snap refreshes do not restart the gateway, so active sandboxes keep running. Restart it to pick up a new version:
+The gateway runs as a system service at `https://127.0.0.1:17670` and reads `/var/snap/openshell/common/gateway.toml`. It requires a client certificate. The install script copies that certificate to the installing user's Snap state and registers the gateway automatically. If you installed with `sudo snap install openshell`, give each trusted user the certificate and register the gateway from that user's account:
 
 ```shell
-sudo systemctl restart snap.openshell.gateway
+d=~/snap/openshell/common/.local/state/openshell/tls
+mkdir -p -m 700 "$d/client"
+sudo cat /var/snap/openshell/common/tls/ca.crt > "$d/ca.crt"
+sudo cat /var/snap/openshell/common/tls/client/tls.crt > "$d/client/tls.crt"
+(umask 077; sudo cat /var/snap/openshell/common/tls/client/tls.key > "$d/client/tls.key")
+chmod 700 "$d" "$d/client"
+chmod 600 "$d/ca.crt" "$d/client/tls.crt" "$d/client/tls.key"
+openshell gateway add https://127.0.0.1:17670 --local --name openshell
+openshell status
 ```
 
+Keep the client key private. Anyone who can read it can authenticate to the local gateway.
+
+On refresh, the Snap replaces any regular gateway config that explicitly enables unauthenticated access or disables TLS with the secure Docker default. Other settings in that file are not carried over. It saves the previous config at `/var/snap/openshell/common/gateway.toml.pre-mtls` with mode `0600`, or at a uniquely suffixed path if that backup already exists. The refresh log prints the actual path. Review the backup before restoring other settings; restoring its insecure settings reopens access.
+
+Snap refreshes restart the gateway to apply the migrated config immediately. This interrupts active sandbox sessions.
+
+If you previously registered the plaintext endpoint, run the certificate and HTTPS registration steps above after the refresh. Remove the old registration first with `openshell gateway remove openshell` if the name already exists. You can also rerun the install script to refresh the registration automatically.
+
 To install a locally built snap, connect its interfaces manually:
 
 ```shell

install.sh

@@ -1295,6 +1295,7 @@ copy_snap_client_bundle() {
   for _file in ca.crt client/tls.crt client/tls.key; do
     as_root cat "${_src}/${_file}" |
       as_target_user sh -c 'umask 077; cat >"$1"' sh "${_dst}/${_file}"
+    as_target_user chmod 600 "${_dst}/${_file}"
   done
 }
 

python/openshell/release_formula_test.py

@@ -155,7 +155,11 @@ def test_snap_wrapper_uses_optional_gateway_config_without_generating_toml() ->
         'export OPENSHELL_DB_URL="${OPENSHELL_DB_URL:-sqlite:${SNAP_COMMON}/gateway.db?mode=rwc}"'
         in wrapper
     )
-    assert 'export OPENSHELL_DISABLE_TLS="${OPENSHELL_DISABLE_TLS:-true}"' in wrapper
+    assert "OPENSHELL_DISABLE_TLS" not in wrapper
+    assert (
+        'export OPENSHELL_LOCAL_TLS_DIR="${OPENSHELL_LOCAL_TLS_DIR:-${SNAP_COMMON}/tls}"'
+        in wrapper
+    )
     assert (
         'exec "${SNAP}/bin/openshell-gateway" --config "$CANONICAL_CONFIG_FILE" "$@"'
         in wrapper

skills/debug-openshell-cluster/SKILL.md

@@ -72,6 +72,7 @@ Common findings:
 - `No active gateway`: register one with `openshell gateway add <endpoint>`.
 - Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active.
 - TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener.
+- A Snap refresh restarts the gateway with its migrated mTLS config. The secure Snap gateway uses `https://127.0.0.1:17670` and requires a client bundle in the user's Snap state. Check `/var/snap/openshell/common/gateway.toml.pre-mtls` for settings replaced during migration, then follow the published Snap installation steps to re-register an old HTTP client.
 - `Unauthenticated` from an edge or OIDC gateway: refresh stored credentials with `openshell gateway login [name]`, then retry. Use `gateway logout` only when intentionally clearing local credentials.
 - A direct development endpoint with a private or self-signed certificate can be isolated with `--gateway-endpoint <url> --gateway-insecure`; do not persist or recommend insecure verification for shared gateways.
 

snap/hooks/install

@@ -2,25 +2,20 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-# Ensure the gateway is secured by default with mTLS.
+# Create the mTLS default and replace insecure configs on refresh.
 
 set -eu
 
 config_file="${SNAP_COMMON}/gateway.toml"
 
-legacy_config='[openshell]
-version = 2
-
-[openshell.gateway]
-
-[openshell.gateway.auth]
-allow_unauthenticated_users = true'
-
 replace=false
 if [ -L "$config_file" ]; then
 	exit 0
 elif [ -f "$config_file" ]; then
-	if [ "$(cat "$config_file")" != "$legacy_config" ]; then
+	# Refresh every insecure default or edited config. Keep secure operator
+	# configs intact, but do not preserve an explicit plaintext or anonymous
+	# access setting merely because the file has other edits.
+	if ! grep -Eq '^[[:space:]]*(allow_unauthenticated_users|disable_tls)[[:space:]]*=[[:space:]]*true([[:space:]#]|$)' "$config_file"; then
 		exit 0
 	fi
 	replace=true
@@ -42,6 +37,17 @@ compute_driver = "docker"
 CONFIG
 
 if [ "$replace" = true ]; then
+	# Keep one private copy for operators who need to restore other settings.
+	backup_file="${config_file}.pre-mtls"
+	backup_tmp=$(mktemp "${backup_file}.XXXXXX")
+	cp "$config_file" "$backup_tmp"
+	chmod 600 "$backup_tmp"
+	if [ ! -e "$backup_file" ] && [ ! -L "$backup_file" ] && ln "$backup_tmp" "$backup_file"; then
+		rm -f "$backup_tmp"
+	else
+		backup_file="$backup_tmp"
+	fi
+	echo "openshell: replacing insecure gateway config; previous config saved at $backup_file" >&2
 	mv -f "$temporary_file" "$config_file"
 	trap - 0 HUP INT TERM
 	exit 0

snapcraft.yaml

@@ -39,15 +39,19 @@ description: |
          sudo cat /var/snap/openshell/common/tls/ca.crt > "$d/ca.crt"
          sudo cat /var/snap/openshell/common/tls/client/tls.crt > "$d/client/tls.crt"
          (umask 077; sudo cat /var/snap/openshell/common/tls/client/tls.key > "$d/client/tls.key")
+         chmod 700 "$d" "$d/client"
+         chmod 600 "$d/ca.crt" "$d/client/tls.crt" "$d/client/tls.key"
          openshell gateway add https://127.0.0.1:17670 --local --name openshell
          openshell status
 
-    After a snap refresh, restart the gateway to pick up the new revision:
-
-         sudo snap restart openshell.gateway
-
-    Restarting the gateway will interrupt active sandbox sessions. The gateway
-    is not restarted automatically to avoid disconnecting running sandboxes.
+    On refresh, configs that explicitly enable plaintext or unauthenticated
+    access are replaced with the secure default. The previous file is saved at
+    /var/snap/openshell/common/gateway.toml.pre-mtls (mode 0600), or a uniquely
+    suffixed path if that backup already exists. The gateway
+    restarts automatically during refresh to apply the new revision and
+    config. This interrupts active sandbox sessions. Users registered against
+    the old HTTP endpoint must remove that gateway registration and repeat
+    step 2, or rerun install.sh.
 
 base: core24
 grade: stable
@@ -95,10 +99,9 @@ apps:
   gateway:
     command: bin/openshell-gateway-wrapper
     daemon: simple
-    # refresh-mode: endure prevents snapd from restarting the gateway daemon
-    # during snap refreshes, which would kill active sandbox sessions.
-    # Operators must manually restart the service after a refresh if needed.
-    refresh-mode: endure
+    # Refresh must activate the migrated mTLS config immediately. This
+    # interrupts active sandbox sessions.
+    refresh-mode: restart
     # Snapd runs this daemon as root. The wrapper serves TLS from the bundle
     # generated in $SNAP_COMMON/tls, and the default config requires client
     # certificates; the installer copies the client bundle to the target

tasks/scripts/test-install-sh.sh

@@ -530,6 +530,9 @@ printf 'ca\n' >"${snap_tls_src}/ca.crt"
 printf 'cert\n' >"${snap_tls_src}/client/tls.crt"
 printf 'key\n' >"${snap_tls_src}/client/tls.key"
 snap_user_home="${tmpdir}/snap-user-home"
+mkdir -p "${snap_user_home}/snap/openshell/common/.local/state/openshell/tls/client"
+printf 'old key\n' >"${snap_user_home}/snap/openshell/common/.local/state/openshell/tls/client/tls.key"
+chmod 644 "${snap_user_home}/snap/openshell/common/.local/state/openshell/tls/client/tls.key"
 (
   as_root() { "$@"; }
   as_target_user() { "$@"; }

tasks/scripts/test-packaging-assets.sh

@@ -124,6 +124,7 @@ if [[ ! -x "$snap_install_hook" ]]; then
   exit 1
 fi
 assert_contains "$snap_install_hook" 'compute_driver = "docker"'
+assert_contains "$snapcraft" 'refresh-mode: restart'
 if [[ ! -x "$(dirname "$snap_install_hook")/post-refresh" ]]; then
   echo "FAIL: Snap post-refresh hook must be executable" >&2
   exit 1

tasks/scripts/test-snap-install-hook.sh

@@ -63,17 +63,69 @@ cp "$legacy" "$common/gateway.toml"
 printf '\n# operator note\n' >>"$common/gateway.toml"
 cp "$common/gateway.toml" "${work}/legacy-edited-before"
 SNAP_COMMON="$common" "$hook"
-cmp -s "${work}/legacy-edited-before" "$common/gateway.toml"
+cmp -s "$expected" "$common/gateway.toml"
+cmp -s "${work}/legacy-edited-before" "$common/gateway.toml.pre-mtls"
+if [[ -z $(find "$common/gateway.toml.pre-mtls" -perm 600) ]]; then
+  echo "FAIL: migrated config backup must be mode 0600" >&2
+  exit 1
+fi
+SNAP_COMMON="$common" "$hook"
+cmp -s "${work}/legacy-edited-before" "$common/gateway.toml.pre-mtls"
+cp "$legacy" "$common/gateway.toml"
+SNAP_COMMON="$common" "$hook"
+cmp -s "$expected" "$common/gateway.toml"
+cmp -s "${work}/legacy-edited-before" "$common/gateway.toml.pre-mtls"
+if [[ $(find "$common" -maxdepth 1 -name 'gateway.toml.pre-mtls.*' -type f | wc -l) -ne 1 ]]; then
+  echo "FAIL: repeated migration must preserve the existing backup" >&2
+  exit 1
+fi
+
+common="${work}/custom-insecure"
+mkdir -p "$common"
+cat >"$common/gateway.toml" <<'EOF'
+[openshell]
+version = 2
+
+[openshell.gateway]
+compute_driver = "docker"
+disable_tls = true # old local override
+
+[openshell.gateway.auth]
+allow_unauthenticated_users = true # old local override
+EOF
+cp "$common/gateway.toml" "${work}/custom-insecure-before"
+SNAP_COMMON="$common" "$hook"
+cmp -s "$expected" "$common/gateway.toml"
+cmp -s "${work}/custom-insecure-before" "$common/gateway.toml.pre-mtls"
+
+common="${work}/custom-secure"
+mkdir -p "$common"
+cat >"$common/gateway.toml" <<'EOF'
+[openshell]
+version = 2
+
+[openshell.gateway]
+compute_driver = "docker"
+# allow_unauthenticated_users = true
+EOF
+cp "$common/gateway.toml" "${work}/custom-secure-before"
+SNAP_COMMON="$common" "$hook"
+cmp -s "${work}/custom-secure-before" "$common/gateway.toml"
+if [[ -e "$common/gateway.toml.pre-mtls" ]]; then
+  echo "FAIL: secure operator config should not be backed up or replaced" >&2
+  exit 1
+fi
 
 common="${work}/post-refresh"
 mkdir -p "$common" "${work}/snap/meta/hooks"
 cp "$hook" "${work}/snap/meta/hooks/install"
-cp "$legacy" "$common/gateway.toml"
+cp "${work}/legacy-edited-before" "$common/gateway.toml"
 SNAP="${work}/snap" SNAP_COMMON="$common" "${hook_dir}/post-refresh"
 if ! cmp -s "$expected" "$common/gateway.toml"; then
-  echo "FAIL: post-refresh hook must migrate the legacy unauthenticated config" >&2
+  echo "FAIL: post-refresh hook must migrate an edited insecure config" >&2
   exit 1
 fi
+cmp -s "${work}/legacy-edited-before" "$common/gateway.toml.pre-mtls"
 
 common="${work}/broken-link"
 mkdir -p "$common"