[Bug] machines.target disabled on RHEL hosts #436

Summary

After a host reboot, the active nspawn worker (kube1) did not start, and the node stayed NotReady for three
days. The agent service started normally and kept logging decision=ReportSucceeded reason="goal state is applied" the whole time, without restarting the worker or reporting a problem.

Environment

• AKS Flex Node v0.3.0 (upgraded from v0.2.0), unbounded v0.21.0 → v0.23.0, AKS 1.37.0
• Host: RHEL 9.8 (kernel 5.14.0-687.66.1.el9_8), systemd 252, offline bootstrap
• Two RHEL hosts, both affected; the Ubuntu 24.04 hosts in the same pool were not rebooted

Root cause (reboot part)

The agent enables the worker under machines.target:

$ systemctl is-enabled machines.target
disabled
$ systemctl is-enabled systemd-nspawn@kube1.service
enabled
$ ls /etc/systemd/system/machines.target.wants/
systemd-nspawn@kube1.service -> /usr/lib/systemd/system/systemd-nspawn@.service

On RHEL the systemd preset leaves machines.target disabled, so it is never reached at boot and the worker
doesn't start (machinectl list → No machines). Ubuntu enables machines.target by default, which is
probably why this doesn't show up there.

Workaround: systemctl enable machines.target, then systemctl start systemd-nspawn@kube1.service.

Agent behavior

daemon-state.json showed "activeMachine": "kube1" while kube1 wasn't running, yet the daemon reported the
goal state as applied every reconcile. Pods on the node stayed Terminating and the node NotReady until the
worker was started manually.

Requests

1. Enable machines.target during bootstrap (or start the active worker from the agent on startup), and add a
   preflight check for it.
2. Have the daemon verify that the active machine is actually running, and restart it or surface a failure
   (node condition, event, or machine status) instead of reporting success.