per-job certificates for job processes - #5263
Conversation
pcnudde
changed the title
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Job processes (SJ/CJ) get a short-lived credential issued at deploy time by a constrained job CA (CA:TRUE pathlen:0, marker extension) provisioned into the server kit by default. Leaf certs keep CN=<site> and carry the job ID; site-scope identity checks reject anything the job CA signed, and cellnet binds a job cert to its job's FQCNs. In secure mode jobs never run on site certificates: missing/expiring job CA fails the deploy, CP rejects a deploy without a credential, job cells refuse to start without one. Docker, K8s and Slurm launchers never ship *.key files to job processes. Closes NVIDIA#5249
cccfbf9 to
e7e382e
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5263 +/- ##
==========================================
+ Coverage 67.40% 67.71% +0.30%
==========================================
Files 1021 1023 +2
Lines 106127 106439 +312
==========================================
+ Hits 71539 72072 +533
+ Misses 34588 34367 -221
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… cells
The K8s workspace-transfer bootstrap cell is named <owner>.ws_transfer_<job_id>,
so the job id is embedded in a segment instead of being one and the job-cert
binding rejected the job pod's own credential at the parent
("claimed endpoint ... is not part of that job"), failing every secure K8s job.
FQCN.belongs_to_job now defines a job's cells as the job cell and its
descendants plus auxiliary cells named <name>_<job_id>; the binding check and
the bootstrap FQCN builder both use it. Adds a cross-process mTLS regression
test that connects a real bootstrap cell to a real parent with a job-bound
cert and verifies another job's cert is still rejected.
The server archived the whole SJ run directory into the job store when a job finished, so download_job returned job_cert/job.crt and job_cert/job.key (found by the Docker e2e validation). Clients kept the CJ credential on disk after the CJ exited. The SP now destroys the SJ credential before the run directory is archived, and the CP destroys the CJ credential as soon as the CJ process has exited. The credential is dead at that point: no renewal, no reuse. Verified with a full POC job: the stored workspace contains no job_cert entries or key files and client run dirs keep the app but no credential.
…_valid_days from server config QA on Slurm found a "cross-job replay" accepted at the site parent and the documented fed_server.json job_cert_valid_days ignored. - Docker and Slurm launchers refuse a clear-text parent connection in secure mode: job-cert binding exists only on mTLS links, and a clear internal listener lets any process that reaches it claim any job FQCN. The shared-file transport is exempt (filesystem permissions). A live test now connects real job cells to a site parent's mTLS internal listener and verifies another job's certificate is rejected there. - SJ and the K8s bootstrap cell pin both TLS roles to the job credential so cellnet's directory-based back-fill can never present a site certificate. - JobRunner reads job_cert_valid_days from the server's fed_server.json / resources.json sections; the SP never loads the application section.
…cess fails The client removed the job credential only after job_handle.wait() returned normally. It is now removed in a finally block, and also when no job handle was registered, so an exceptional wait or a launch that never produced a handle cannot leave the credential on disk.
…ger hooks - _deploy_job in secure mode with a real job CA: SJ credential written to the run dir (CN of the server cert, job id, leaf + job CA chain, key 0600) and the CJ credential carried in the per-site deploy message. - FLServerStarterConfiger / FLClientStarterConfiger point the job process at the job credential when one exists, and leave site paths alone otherwise.
…owner-relative job segment Review follow-ups on NVIDIA#5263: - grpc_driver.py / aio_grpc_driver.py called add_grpc_peer_job_id without importing it, so every authenticated gRPC stream failed with NameError. Both servicers now import it; new tests drive their authenticated stream paths with site and job certificates. setup.cfg ignores F821, which is why flake8 never reported the undefined name. - FQCN.belongs_to_job matched the job id in any segment, so job A's cert was accepted on site-1.<job-B>.ws_transfer_<job-A>. The job segment must now be the one right after the owner's prefix, which the identity resolver reports alongside the expected identity (resolve_owner). Covered in the resolver, handshake, certificate-cache and live mTLS tests. - The live tests asserted the child's transient connected flag, which can read True before the parent rejects the handshake. They now assert that an application request fails and that the parent logged the rejection.
# Conflicts: # nvflare/app_opt/job_launcher/slurm/batch.py
|
This PR adds job_id handling in F3, which is the wrong layer. Here's how the network is currently structured: SFM (including the transport drivers) is the bottom layer. It sends messages from endpoint to endpoint and has no concept of cells — it cannot forward messages between them. Cellnet sits above SFM. It introduces FQCNs and knows how to relay messages between non-directly-connected cells using the FQCN hierarchy. It understands dot-separated FQCN components, but it has no concept of jobs. That's intentional — the client/server layer above Cellnet is responsible for incorporating job_id and constructing the correct FQCN so messages are properly relayed. job_id belongs at that higher layer, not in Cellnet or below. Keeping this separation is important: Cellnet is a general-purpose communication layer that other applications can reuse. Leaking job concepts into it breaks that abstraction. |
The live tests generated job ids with uuid4() at import time, so each pytest-xdist worker collected different parametrized test ids and CI failed with "Different tests were collected". Use fixed ids.
|
Agreed: job_id does not belong in F3. Proposal to fix the layering while keeping the check at the handshake, where it has to happen. What stays in F3, job-free
What moves up to
Extension: new non-critical OID Tests: the cellnet unit and live tests become scope tests without job vocabulary; the job-layer tests assert the issuer emits the right scope. Runtime behavior is unchanged, so the K8s/Docker/Slurm validation results stay valid. Net effect: F3 knows that a certificate can be restricted to FQCN subtrees, nothing about jobs. The alternative is an authorizer callback that the FL layer installs per cell, which keeps cellnet free of any extension parsing but relies on every cell creator (SP, CP, relay, bootstrap) remembering to install it; I think that is the worse trade. If this works for you I will rework the PR accordingly. |
|
@nvidianz See my proposal above. I also want to remove the custom OID that is used in the certificate. It should be fine, but I do not want to go through the registration steps if I can avoid it. |
|
Amendment to the proposal above: no new OID. In fact, no private OIDs at all. This PR will do the same and drop all three custom extensions:
Reader rules, same as the SSO grants: exact scheme/host/prefix match, URIs on other hosts ignored, malformed URIs under the root rejected. FQCN characters are all URL-unreserved, so prefixes need no encoding, but they are percent-encoded anyway for a uniform fail-closed parser. The The rest of the proposal stands: cellnet only learns that a certificate may restrict which FQCN subtrees it can claim, the FL layer decides the prefixes when it issues the credential, and F3 carries no job vocabulary. The external-CA recipe gets simpler too: |
…s as URI SANs Review follow-up on NVIDIA#5263: job_id belongs above cellnet, and private OIDs under NVIDIA's enterprise arc cannot be allocated. F3 now knows only that a certificate may restrict the cells it can claim: drivers expose the authenticated peer certificate (PEER_CERT), and CellIdentityResolver rejects a claimed FQCN outside the certificate's cell-scope URIs (FQCN.is_ancestor). PEER_JOB_ID, the job-id helpers in net_utils, FQCN.belongs_to_job/job_aux_name and resolve_owner are gone. Certificate attributes are https URI Subject Alternative Names under https://nvidia.com/nvflare/v1/ (nvflare/fuel/sec/cert_uri.py): job/<id> and cell/<fqcn> on job leaves, ca/job on the job CA. The three private OIDs are removed. The FL layer decides the cells when it issues a credential: the job cell and the bootstrap cell under the owning CP or server FQCN, which also covers clients behind relays.
|
Thanks. I agree that we should avoid private OIDs. I recommend using a standard field in the certificate subject DN instead of introducing NVFlare URI SAN semantics. Of the standard DN fields,
The reader should collect all For completeness, the common DN fields and their ASN.1 OIDs are:
|
# Conflicts: # nvflare/lighter/impl/cert.py
|
Thanks, agreed on dropping the private OIDs; that is what the current head does. I looked at moving the three values into repeated 1. Issuer-name coupling. A marker in the job CA's subject DN becomes part of every leaf's issuer name. The issuer today rebuilds the issuer name from the CA's CN; with 2. The 64-character bound on
URI SANs have neither issue: no length cap, no effect on the issuer name, and SAN is the extension X.509 defines for carrying additional identities of the subject. Both encodings are protected by the signature, and a prefix convention inside I propose to keep the URI SANs. |
Review follow-up on NVIDIA#5263: the per-driver steps that derive PEER_CN from the peer certificate and record PEER_CERT were duplicated across the socket, asyncio, websocket and both gRPC transports, each with its own fallback convention, and pulled nvflare.fuel.hci.security into the transport layer. Connection.record_peer (F3 base class) now takes the raw peer certificate, DER or PEM as the transport provides it, and sets PEER_CERT and PEER_CN once; a secure connection with no peer certificate still reports PEER_CN "N/A". Each driver hands over the certificate in one line. Consumers of PEER_CN and PEER_CERT are unchanged. Driver files are now smaller than upstream.
3 participants
Summary
Closes #5249 (parent overview: #5246).
Job processes (SJ/CJ) run on the site's long-lived certificate and private key, so any job-supplied code can read the site key and impersonate the site indefinitely. This PR gives every job process its own short-lived credential, makes it the only credential a secure job can run on, and keeps the site keys out of job processes.
CertBuildergenerates a job-signing intermediate CA (job_ca.crt/job_ca.key,CA:TRUE pathlen:0, root-signed marker URI SAN) into the server startup kit. On by default;enable_job_ca: falseopts out. Persisted incert.json, regenerated if expired. A customer-supplied root passed tonvflare provisionis covered. The distributednvflare cert/nvflare packageflow is not (no root key at packaging time, so no job CA, so no secure jobs); it is addressed separately.CN=<site>, URI SANs naming the job and the cells it may claim,notBeforebackdated for clock skew, validityjob_cert_valid_daysfrom the server'sfed_server.jsonorresources.json,--set, orNVFLARE_JOB_CERT_VALID_DAYS, default 30 days, clamped to the CA). The SJ credential is written to the run dir after app deployment; each CJ credential is pushed in its site's deploy message over the existing authenticated CP-SP channel.job_deploy_detail; the CP rejects a deploy request without a valid job credential; a job cell refuses to start without one; launchers refuse to launch a secure job that has no credential. Non-secure mode and the simulator use no certificates and are unaffected.ssl_cert/ssl_private_keypoint at the job credential, so nothing in SJ/CJ refers to the site key. Job cells (and the Kubernetes bootstrap cell) use it for TLS and message-level crypto in both TLS roles, so cellnet's directory-based credential back-fill can never present a site certificate. The startup content-integrity check no longer requires the site key in job processes.IdentityVerifier.verify_common_name()rejects any certificate carrying the job URI and any chain containing the job-CA marker URI, so neither a leaked job leaf key nor a stolen job CA key can register a site, log in as admin, or pass as the server.stcp/mtls; the shared-file transport is exempt). Kubernetes job pods already use the parent's externalstcplistener with mTLS. A live test connects real job cells to a site parent's mTLS internal listener and verifies another job's certificate is rejected.download_jobnever contains it; the CP destroys the CJ credential as soon as the CJ process exits. The credential is dead at that point (no renewal, no reuse).https://nvidia.com/nvflare/v1/(job/<job id>andcell/<fqcn>on job leaves,ca/jobon the job CA;nvflare/fuel/sec/cert_uri.py). A private extension would need an OID under NVIDIA's enterprise arc, which cannot be allocated, and2.25OIDs break Go parsers; a domain-owned URI needs no registry. Readers match the root exactly, ignore other hosts, and fail closed on malformed URIs.PEER_CERT);CellIdentityResolverenforces one generic rule, a certificate carrying cell-scope URIs may only claim an FQCN equal to or under one of those cells, at the connection handshake and on the certificate cached for message-level crypto. Which cells a job credential lists is decided where it is issued: the job cell and the workspace-transfer bootstrap cell under the owning CP or server FQCN (which covers clients behind relays).site-1.<job B>.ws_transfer_<job A>is job B's cell and is rejected for job A's certificate. Live cross-process mTLS tests cover a server root, a site parent's internal listener, and the bootstrap cell.*.key(includingjob_ca.key): Docker binds the startup kit file by file; Kubernetes drops keys from the startup Secret and delivers the job credential through the per-pod credential Secret (NVFLARE_JOB_CERT/NVFLARE_JOB_KEY), installed by the job process before its bootstrap cell starts, withjob_cert/excluded from workspace bundles and result uploads; Slurm (apptainer/pyxis) binds a keyless staged copy of the kit. The in-process launcher and Slurmsandbox: noneshare the host filesystem and cannot isolate.docs/design/per_job_certs_design.md; user guidedocs/user_guide/admin_guide/security/per_job_certificates.rst(incl. openssl recipe for an externally issued job CA).Compatibility
Existing secure-mode projects must be re-provisioned (the root CA is reused; only
job_ca.*is added to the server kit) before jobs will deploy. Server and clients must run the same release: an older client would run its job process on site certificates, which secure mode no longer allows. Non-secure mode and the simulator are unchanged.