[BUG-NOK]: Device onboarding not successful after micro OS download

[lokesh0013]

Bug Description

after registering a device and booting it with the efi file the device gets a ip and downloads the micro os normally. after the download the UI is not reflected with the onboarded stated it remains in unkown state.

debuging more we found the devices which where onboarded before are in no connection as the agents are throwing a certifitcate issue and common on all the devices with are onboarded on this emf.

level=error msg="Failed to send update status: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"" component="Update Agent" version=1.5.2-5680f0b
Jun 04 05:26:16 host-629d6fd1 platform-update-agent[782]: time="2026-06-04T05:26:16Z" level=error msg="will try to reconnect because of failure: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"" component="Update Agent" version=1.5.2-5680f0b

futher debuging from backend found that All apps are in sync and only root-app is in Unknown state with cert error.

Software Version

3.1

Operating System

Ubuntu 22.04 (LTS)

System Setup

a edge device registered and ready to onbaord with the usb.

Reproducible Steps

register a device on EMF 3.1 and onboard it with a usb

[amathan]

The issue on the argocd resolved once the new certificate was added to Repository certificates on argo settings and root-app is healthy.

How the EMF services get to know about new certificate and get itself updated? Currently we see onboarding of device is failing.

[niket-intc]

Can you clarify these and provide us logs for further analysis.

1. Did the Orchestrator certificate expired? The orchestrator certificates are generated for 1 year.

2. Did the boot-ca certificate was also updated.

3. Are all the agents and services(pods) are in running state post certificate update in argocd?
   Currently onboarded edgenode do not support update of root-ca certificate when certificates are updated in orchestrator.

4. Restart the dkam and onboarding-manager pod and wait till the curation in dkam is completed.
   Check for this logs
   {"level":"info","component":"InfraDKAMMain","InfraSec":"true","caller":"github.com/open-edge-platform/infra-onboarding/dkam/cmd/dkammgr/main.go:190","timestamp":"2026-06-04T11:38:02.858965584Z","message":"Signed MicroOS and moved to PVC"}
   Share the logs of onboarding-manager, dkam and tink-worker while onboarding the edgenode to further investigate the issue.

[amathan]

1. Yes, the certificate was expired yesterday and caused us this issue, and we could see few of the cert's validity is for 3 months only and in case of tinker is more.

Below is an example of newly brought up server which has validity for 3 months for keycloak whereas for tinker its 1yr

2. No, we dint update any other certificate other than gitea repository certificates in argocd.

3)Yes, for one of the edges after manual updating of cert, agents started to communicate to orchestrator and become online. We feel it's not co related to argocd. [Argocd repository certificate updating is meant for only gitea communication to succeed]

1895.zip

[amathan]

@niket-intc Any updates ? Hope the above information is suffice. Pls let us know if any other info is needed.

[niket-intc]

@niket-intc Any updates ? Hope the above information is suffice. Pls let us know if any other info is needed.

Try these steps when certificate is updated.

1. Disable sync for the following ArgoCD applications:

   • root-app
   • nginx-ingress-pxe-boots

2. Update Certificate

   • Update the certificate in the nginx-ingress-pxe-boots application:
     • Certificate Name: tls-boots
     • Key Size: Change RSA key size from 4096 to 3072
       Ensure the certificate is generated or updated with the correct key size before proceeding.

3. Delete Secret and restart DKAM Pod
   Run the following commands to delete the old secrets and restart the DKAM pod:
   kubectl delete secret tls-boots -n orch-boots
   kubectl delete secret boots-ca-cert -n orch-gateway
   kubectl delete secret boots-ca-cert -n orch-infra
   kubectl delete pod dkam-599ff7f9dc-ptxwf -n orch-infra

Wait approximately 5 minutes to allow DKAM and related components to reload and reflect the new certificate.

[amathan]

Steps performed:

1. Disabled auto-sync for root-app and nginx-ingress-pxe-boots from argocd Gui
2. kubectl edit certificate tls-boots -n orch-boots // Change RSA key size from 4096 to 3072
3. kubectl delete secret tls-boots -n orch-boots
4. kubectl delete secret boots-ca-cert -n orch-gateway
5. kubectl delete secret boots-ca-cert -n orch-infra
6. kubectl delete pod dkam-<> -n orch-infra
   After doing these changes and we tried to onboard a fresh device and it dint succeed.

Additionally, App deployment for an existing device gives

Get "https://gitea.prod05.mxgmanager.com/apporch/83bf658f-1b29-46ba-a074-d57ce4d9c5e0.git/info/refs?service=git-upload-pack%22: tls: failed to verify certificate: x509: certificate signed by unknown authority

[amathan]

@niket-intc Any workaround possible to handle such situations?

[lokesh0013]

Hi, The connect-agent pod is unable to establish a connection to the MXG Gateway because the JWT authentication token used by the agent has expired. The gateway rejects the connection request with HTTP 400 Bad Request and returns the error "token has invalid claims: token is expired", which causes a WebSocket handshake failure (websocket: bad handshake) and results in the pod repeatedly crashing and restarting.

[amathan]

Any updates on this @niket-intc