per-job certificates for job processes

docs/design/job_launcher_and_job_handle.md

@@ -414,7 +414,7 @@ Launch sequence:
 | 1 | Sanitize job ID via `uuid4_to_rfc1123`. Extract `site_name`, `job_image` from `get_job_launcher_spec(job_meta, site_name, "k8s")`. Raise if `WORKSPACE_OBJECT` missing. |
 | 2 | Read `JOB_PROCESS_ARGS`; raise if absent or `EXE_MODULE` missing. If `<workspace>/local/study_runtime.yaml` exists, parse it (strict v2) and resolve the job study's datasets, env, secret_env, secret_mounts, container name, and pod template from it; coexistence with a v1 study data file is a hard error. Otherwise resolve dataset PVC mounts from `study_data_pvc_file_path` when configured and the YAML file contains entries for the job study. |
 | 3 | Build `job_config`: name, image, args from `get_module_args()`. Use `launcher_spec[site][k8s].python_path` for the pod command when present, falling back to `default_python_path`. Mount the job workspace at `workspace_mount_path`, mount the startup-kit Secret at `<workspace_mount_path>/startup`, and set custom-code `PYTHONPATH` under `workspace_mount_path`. Add the workspace `emptyDir.sizeLimit` and `resources.requests/limits["ephemeral-storage"]` from `launcher_spec[site][k8s].ephemeral_storage` when present, falling back to the launcher default. Add K8s CPU and memory limits from `launcher_spec`; add GPU limits from the flat `resource_spec[site].num_of_gpus` GPU resource requirement. Apply `launcher_spec[site][k8s].pending_timeout` when present. Missing study entries skip data PVC mounts and log a warning. If a pod template is resolved, preserve template pod fields and sidecars while replacing NVFlare-owned fields such as pod name, job container image/command/args, workspace mounts, transfer env vars, image pull secrets, and resources. |
-| 3.5 | Create the per-job credential Secret `nvflare-cred-<pod_name>` (bootstrap credentials plus the workspace transfer token) and reference it from the job container via `env[].valueFrom.secretKeyRef` — credential values never appear in the pod object. |
+| 3.5 | Create the per-job credential Secret `nvflare-cred-<pod_name>` (bootstrap credentials, the workspace transfer token, and — when the job has one — the per-job certificate and key as `NVFLARE_JOB_CERT` / `NVFLARE_JOB_KEY`) and reference it from the job container via `env[].valueFrom.secretKeyRef` — credential values never appear in the pod object. The startup Secret never includes `*.key`; in secure mode a job without a credential is refused. |
 | 4 | Create `K8sJobHandle` (carries the credential Secret name). |
 | 5 | `core_v1.create_namespaced_pod()`, then patch the credential Secret with an ownerReference to the created pod (GC backstop). On any exception: delete the credential Secret if the pod was never created, set `terminal_state = TERMINATED`, preserve `EXCEPTION` as the return code, and return handle. |
 | 6 | `job_handle.enter_states([RUNNING])`. On any `BaseException`: `terminate()` then re-raise. |

docs/design/no_shared_pvc_k8s_workspace_design.md

@@ -55,9 +55,14 @@ For each launched job pod, the launcher creates a pod manifest with:
 - optional study-data PVC mounts at `/data/<study>/<dataset>`
 
 The launcher also creates or updates a startup Secret for the participant site.
-That Secret contains the startup-kit files needed by the launched process, such
-as certificates, keys, and JSON config files, and those files appear in the
-pod under `/var/tmp/nvflare/workspace/startup` via the Secret mount.
+That Secret contains the startup-kit files needed by the launched process —
+certificates, `rootCA.pem`, and JSON config files — and those files appear in
+the pod under `/var/tmp/nvflare/workspace/startup` via the Secret mount. Site
+private keys (`*.key`) are never included: the job's own certificate and key
+travel in the per-job credential Secret (`NVFLARE_JOB_CERT` /
+`NVFLARE_JOB_KEY`) and the job process writes them into its run directory
+before creating the bootstrap cell. In secure mode the launcher refuses to
+start a job that has no credential (see `per_job_certs_design.md`).
 
 ## Transfer Architecture
 
@@ -118,7 +123,8 @@ The download sequence is:
    `NVFL_WORKSPACE_TRANSFER_TOKEN`.
 2. It creates a short-lived bootstrap child cell using the startup kit and the
    existing parent connection settings. The bootstrap FQCN is
-   `<owner_fqcn>.ws_transfer_<job_id>`. When the child process is a client
+   `<owner_fqcn>.ws_transfer_<job_id>`, one of the cells the job credential lists
+   as its own, so the certificate is accepted there. When the child process is a client
    worker, the bootstrap cell reuses that worker's `client_name`, auth token,
    token signature, and `ssid`. When the child process is a server runner, the
    bootstrap cell uses the same server-job auth identity as the main runner.
@@ -203,8 +209,9 @@ artifacts.
 The key protections are:
 
 - `startup/` is mounted read-only from a Kubernetes Secret
-- secure mode bootstrap cells use `rootCA.pem` plus the available startup cert
-  and key pair
+- secure mode bootstrap cells use `rootCA.pem` plus the job credential the
+  launcher delivered; there is no fallback to the site key
+- workspace bundles and result uploads never include `job_cert/`
 - the launcher passes the parent listener's connection-security setting into
   the child process args, and the bootstrap cell installs its CellNet auth
   headers before `cell.start()` so the parent accepts the initial registration