Add experimental flex-operator Helm chart

CHANGELOG.md

@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Added
 
+- Experimental `charts/flex-operator` Helm chart that packages the Unbounded operator (v0.8.0) for installation as an AKS cluster extension, with `make chart-check` to detect drift from the upstream manifests.
 - Support for Azure Container Linux and other hosts with a read-only `/usr`, and a Butane config, `scripts/aks-flex-node-bootstrap.bu`, that bootstraps a host provisioned by Ignition ([#323](https://github.com/Azure/AKSFlexNode/pull/323)).
 
 ### Changed

Makefile

@@ -125,6 +125,13 @@ docs-check: docs-cli-check
 	@echo "Checking documentation content..."
 	@python3 hack/docs/check_docs.py
 
+.PHONY: chart-check
+chart-check: ## Lint the flex-operator chart and check it against the pinned upstream manifests
+	@command -v helm >/dev/null || { echo "helm is required for chart-check" >&2; exit 1; }
+	@helm lint --strict charts/flex-operator --namespace unbounded-system
+	@helm template flex-operator charts/flex-operator --namespace unbounded-system \
+		| python3 hack/flex-operator/compare_upstream.py - hack/flex-operator/upstream-v0.8.0
+
 .PHONY: check
 check: docs-check fmt-all vet lint test
 	@echo "All checks passed!"

charts/flex-operator/.helmignore

@@ -0,0 +1,3 @@
+.DS_Store
+*.swp
+*.bak

charts/flex-operator/Chart.yaml

@@ -0,0 +1,15 @@
+apiVersion: v2
+name: flex-operator
+description: >-
+  PoC AKS cluster extension that installs the upstream Unbounded operator
+  (unbounded-operator) as the "Flex operator" for flex nodes for AKS.
+type: application
+version: 0.1.0
+appVersion: "v0.8.0"
+home: https://github.com/Azure/unbounded
+sources:
+  - https://github.com/Azure/unbounded/tree/v0.8.0/deploy/unbounded-operator
+annotations:
+  # Keep in sync with hack/flex-operator/upstream-v0.8.0 and UNBOUNDED_VERSION
+  # in docs/usage/getting-started.md.
+  unbounded.azure.com/upstream-manifests: deploy/unbounded-operator@v0.8.0

charts/flex-operator/README.md

@@ -0,0 +1,81 @@
+# flex-operator Helm chart
+
+> [!NOTE]
+> Experimental. This chart packages the Unbounded operator so that AKS can
+> install and upgrade it as a managed cluster extension. It isn't a supported
+> installation method yet; use `kubectl unbounded install` as described in the
+> [getting-started guide](../../docs/usage/getting-started.md).
+
+The chart wraps the upstream
+[`deploy/unbounded-operator`](https://github.com/Azure/unbounded/tree/v0.8.0/deploy/unbounded-operator)
+manifests at the release pinned in `Chart.yaml` (`appVersion`). It installs
+the same ServiceAccount, ConfigMap, ClusterRole, ClusterRoleBinding, and
+Deployment that `kubectl unbounded install` applies. The operator then installs
+its CRDs at startup and reconciles components from Sites.
+
+## Install
+
+The chart must be installed into the `unbounded-system` namespace; rendering
+into any other namespace fails.
+
+```bash
+helm install flex-operator charts/flex-operator \
+  --namespace unbounded-system --create-namespace --wait
+kubectl -n unbounded-system rollout status deploy/unbounded-operator
+```
+
+## Values
+
+| Value | Default | Description |
+| --- | --- | --- |
+| `operator.imageRegistry` | `ghcr.io/azure` | Image prefix for the operator and every operator-managed component. |
+| `operator.image` | `unbounded-operator:<appVersion>` | Operator `name:tag` without a registry. |
+| `operator.imagePullPolicy` | `IfNotPresent` | The tag is pinned, so the upstream `Always` isn't needed. |
+| `operator.apiServerEndpoint` | `""` | API server endpoint advertised to remote machines. Empty lets the operator discover it. |
+| `operator.reapLegacyResources` | `true` | Upstream legacy-migration switch. |
+| `sites`, `sitePeering` | empty | Optional Sites and SitePeering applied by a post-install/post-upgrade Job. |
+
+The Job creates the same Site objects as `kubectl unbounded site init`
+(Machina enabled by default; set `machina: false` per Site to disable). Any Site
+makes the operator install Unbounded-Net, which replaces the cluster CNI. Set
+`sites` only on an AKS cluster created with `--network-plugin none`. See
+[values.yaml](values.yaml) for an example. The Job runs `kubectl` directly
+because the default image has no shell.
+
+## Differences from the upstream manifests
+
+| Area | Upstream | Chart |
+| --- | --- | --- |
+| Namespace | Rendered as an object | Release namespace, must be `unbounded-system` |
+| Operator image | Stamped by `kubectl unbounded install` | `operator.imageRegistry` and `operator.image` |
+| `imagePullPolicy` | `Always` | `IfNotPresent` |
+| Config hash annotation | Hash of the CLI's JSON | Hash of the rendered ConfigMap data |
+| Sites | `kubectl unbounded site init` | Optional `sites` value |
+
+Check for unintended drift after changing the chart or the pinned release:
+
+```bash
+helm template flex-operator charts/flex-operator -n unbounded-system \
+  | python3 hack/flex-operator/compare_upstream.py - hack/flex-operator/upstream-v0.8.0
+```
+
+## Upgrade
+
+Upgrading the chart to a newer Unbounded release upgrades the operator; the
+operator then upgrades its CRDs and every component to its own version. Sync
+`templates/rbac.yaml` with the new release's
+`deploy/unbounded-operator/02-rbac.yaml.tmpl` and regenerate
+`hack/flex-operator/upstream-<version>` so `make chart-check` stays green.
+
+## Uninstall
+
+`helm uninstall` removes the operator, its RBAC, and its ConfigMap. It leaves
+the CRDs the operator installed, any Sites, and the component workloads that
+the operator reconciled. Delete Flex node pools and Sites before uninstalling.
+
+## Open questions
+
+- Who owns CRD, Site, and component cleanup when the operator is removed.
+- Whether Sites belong in this chart or remain with `kubectl unbounded site init` and AKS.
+- How to detect and adopt an operator installed with `kubectl unbounded install`.
+- Image mirroring for the operator and every component under one registry prefix.

charts/flex-operator/templates/NOTES.txt

@@ -0,0 +1,8 @@
+Flex operator (unbounded-operator {{ .Chart.AppVersion }}) installed in {{ .Release.Namespace }}.
+The operator installs Unbounded CRDs at startup; check:
+  kubectl -n {{ .Release.Namespace }} rollout status deploy/unbounded-operator
+{{- if not .Values.sites }}
+No Sites were configured. Create Sites (kubectl unbounded site init, or set
+.Values.sites on a --network-plugin none cluster) before attaching flex nodes.
+{{- end }}
+Uninstall does not delete operator-installed CRDs, Sites or component workloads.

charts/flex-operator/templates/_helpers.tpl

@@ -0,0 +1,46 @@
+{{/*
+Upstream components and the CLI assume unbounded-system. Install the extension
+with --release-namespace unbounded-system; any other namespace fails render.
+*/}}
+{{- define "flex-operator.namespace" -}}
+{{- if ne .Release.Namespace "unbounded-system" -}}
+{{- fail (printf "flex-operator must be installed into release namespace unbounded-system, got %q" .Release.Namespace) -}}
+{{- end -}}
+{{- .Release.Namespace -}}
+{{- end -}}
+
+{{- define "flex-operator.labels" -}}
+app.kubernetes.io/name: unbounded-operator
+app.kubernetes.io/component: controller
+app.kubernetes.io/part-of: flex-operator
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }}
+{{- end -}}
+
+{{- define "flex-operator.image" -}}
+{{- $nameTag := default (printf "unbounded-operator:%s" .Chart.AppVersion) .Values.operator.image -}}
+{{- printf "%s/%s" (trimSuffix "/" .Values.operator.imageRegistry) $nameTag -}}
+{{- end -}}
+
+{{- define "flex-operator.configData" -}}
+UNBOUNDED_API_SERVER_ENDPOINT: {{ .Values.operator.apiServerEndpoint | quote }}
+UNBOUNDED_IMAGE_REGISTRY: {{ trimSuffix "/" .Values.operator.imageRegistry | quote }}
+UNBOUNDED_REAP_LEGACY_RESOURCES: {{ .Values.operator.reapLegacyResources | toString | quote }}
+{{- end -}}
+
+{{/*
+Fail when an operator Deployment exists that this release doesn't own, for
+example one installed by `kubectl unbounded install`. Helm would otherwise
+fail later with a less clear ownership error, or a second release would fight
+the first. lookup returns nothing during `helm template`, so this only runs
+against a live cluster.
+*/}}
+{{- define "flex-operator.assertNoForeignOperator" -}}
+{{- $existing := lookup "apps/v1" "Deployment" .Release.Namespace "unbounded-operator" -}}
+{{- if $existing -}}
+{{- $owner := dig "metadata" "annotations" "meta.helm.sh/release-name" "" $existing -}}
+{{- if ne $owner .Release.Name -}}
+{{- fail (printf "unbounded-operator already exists in %s and isn't managed by release %q (managed by %q). Remove the existing installation, or adopt it explicitly, before installing flex-operator." .Release.Namespace .Release.Name (default "kubectl unbounded install or another tool" $owner)) -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}

charts/flex-operator/templates/configmap.yaml

@@ -0,0 +1,10 @@
+# Mirrors upstream 03-configmap.yaml.tmpl. Exposed to the operator via envFrom.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: unbounded-operator-config
+  namespace: {{ include "flex-operator.namespace" . }}
+  labels:
+    {{- include "flex-operator.labels" . | nindent 4 }}
+data:
+  {{- include "flex-operator.configData" . | nindent 2 }}

charts/flex-operator/templates/deployment.yaml

@@ -0,0 +1,78 @@
+# Mirrors upstream 04-deployment.yaml.tmpl at the pinned release.
+{{- include "flex-operator.assertNoForeignOperator" . }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: unbounded-operator
+  namespace: {{ include "flex-operator.namespace" . }}
+  labels:
+    {{- include "flex-operator.labels" . | nindent 4 }}
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: unbounded-operator
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: unbounded-operator
+        app.kubernetes.io/component: controller
+      annotations:
+        kubernetes.azure.com/set-kube-service-host-fqdn: "true"
+        unbounded-cloud.io/operator-config-hash: {{ include "flex-operator.configData" . | sha256sum | quote }}
+    spec:
+      # The operator installs the cluster CNI, so it must reach the API server
+      # before pod networking or cluster DNS exists.
+      hostNetwork: true
+      dnsPolicy: Default
+      serviceAccountName: unbounded-operator
+      tolerations:
+        - key: node.kubernetes.io/not-ready
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: controller
+          image: {{ include "flex-operator.image" . }}
+          imagePullPolicy: {{ .Values.operator.imagePullPolicy }}
+          args:
+            - --leader-elect=true
+            - --leader-elect-namespace={{ include "flex-operator.namespace" . }}
+            - --namespace={{ include "flex-operator.namespace" . }}
+          envFrom:
+            - configMapRef:
+                name: unbounded-operator-config
+          env:
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          ports:
+            - name: metrics
+              containerPort: 8080
+              protocol: TCP
+            - name: health
+              containerPort: 8081
+              protocol: TCP
+          resources:
+            {{- toYaml .Values.operator.resources | nindent 12 }}
+          startupProbe:
+            # CRD bootstrap (up to 4m) runs before /healthz is served.
+            httpGet:
+              path: /healthz
+              port: health
+            periodSeconds: 5
+            failureThreshold: 54
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          readinessProbe:
+            httpGet:
+              path: /readyz
+              port: health
+            initialDelaySeconds: 5
+            periodSeconds: 5

charts/flex-operator/templates/rbac.yaml

@@ -0,0 +1,114 @@
+# Copied verbatim from upstream deploy/unbounded-operator/02-rbac.yaml.tmpl
+# (v0.8.0). The operator installs component RBAC it does not itself hold,
+# hence escalate/bind. Review before production.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: unbounded-operator
+  labels:
+    app.kubernetes.io/name: unbounded-operator
+    app.kubernetes.io/component: controller
+rules:
+  - apiGroups: ["unbounded-cloud.io"]
+    resources: ["sites"]
+    verbs: ["get", "list", "watch", "create"]
+  - apiGroups: ["unbounded-cloud.io"]
+    resources: ["sites/status"]
+    verbs: ["get", "patch", "update"]
+  # Legacy net-group Sites are read during migration and translated into the
+  # machina-group Sites above; their finalizers are cleared (update/patch) so the
+  # CRD can be deleted once drained.
+  - apiGroups: ["net.unbounded-cloud.io"]
+    resources: ["sites"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: ["net.unbounded-cloud.io"]
+    resources: ["sitenodeslices"]
+    verbs: ["get", "list", "watch"]
+  # Cluster-scoped custom resources whose embedded secret references are
+  # repointed from the legacy namespaces to unbounded-system.
+  - apiGroups: ["unbounded-cloud.io"]
+    resources: ["machines", "machineoperationcredentials"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
+  - apiGroups: [""]
+    resources: ["namespaces", "serviceaccounts", "configmaps", "services", "secrets"]
+    verbs: ["get", "list", "watch", "create", "patch", "update", "delete", "deletecollection"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["pods", "replicationcontrollers", "persistentvolumeclaims"]
+    verbs: ["get", "list", "watch"]
+  # Read-only, and only ever read: a component that registers admission
+  # webhooks or an aggregated API points them at a Service, and a registration
+  # whose backend is not serving is worse than none. Endpoints answer whether
+  # anything is actually behind that Service. The net controller Service has no
+  # selector - its leader pod writes these itself - so there is no other way to
+  # tell a running backend from a Deployment that merely exists.
+  #
+  # The verbs are exactly what the reads need and no more. These go through
+  # Env.LiveReader, which bypasses the manager cache. Both endpoint objects use
+  # the Service's own name, so get suffices. Neither is watched, which is the
+  # point of reading them live.
+  - apiGroups: [""]
+    resources: ["endpoints"]
+    verbs: ["get"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
+    verbs: ["get"]
+  - apiGroups: ["apps"]
+    resources: ["deployments", "daemonsets"]
+    verbs: ["get", "list", "watch", "create", "patch", "update", "delete", "deletecollection"]
+  - apiGroups: ["apps"]
+    resources: ["statefulsets", "replicasets"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["batch"]
+    resources: ["jobs", "cronjobs"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["scheduling.k8s.io"]
+    # The gantry component installs a low-priority PriorityClass (gantry-low)
+    # so its agents are preemptible under cluster pressure.
+    resources: ["priorityclasses"]
+    verbs: ["get", "list", "watch", "create", "patch", "update"]
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
+    # escalate + bind let the operator install the component RBAC (net,
+    # machina, metalman, storage) whose Roles/ClusterRoles grant permissions
+    # the operator itself does not hold; without them the apiserver rejects the
+    # role creation as a privilege escalation. deletecollection lets the reaper
+    # DeleteAllOf the legacy component RBAC by label.
+    verbs: ["get", "list", "watch", "create", "patch", "update", "delete", "deletecollection", "escalate", "bind"]
+  - apiGroups: ["admissionregistration.k8s.io"]
+    resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations", "validatingadmissionpolicies", "validatingadmissionpolicybindings"]
+    verbs: ["get", "list", "watch", "create", "patch", "update"]
+  - apiGroups: ["apiregistration.k8s.io"]
+    resources: ["apiservices"]
+    verbs: ["get", "list", "watch", "create", "patch", "update"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "list", "watch", "create", "patch", "update"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create", "patch"]
+  - apiGroups: ["events.k8s.io"]
+    resources: ["events"]
+    verbs: ["create", "patch", "update"]
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: unbounded-operator
+  labels:
+    app.kubernetes.io/name: unbounded-operator
+    app.kubernetes.io/component: controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: unbounded-operator
+subjects:
+  - kind: ServiceAccount
+    name: unbounded-operator
+    namespace: {{ include "flex-operator.namespace" . }}

charts/flex-operator/templates/serviceaccount.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: unbounded-operator
+  namespace: {{ include "flex-operator.namespace" . }}
+  labels:
+    {{- include "flex-operator.labels" . | nindent 4 }}