🌱 spokes still use per-host certs - wildcard exists but nothing serves it (oke + lke)

[clubanderson]

Summary

A fleet wildcard certificate exists and is valid, but no spoke actually uses it. Every spoke ingress still carries its own tls: block pointing at a per-namespace hive-tls-hc secret, and an explicit tls: block always wins over the controller's --default-ssl-certificate. So the wildcard is currently inert, and the fleet is still minting one certificate per host against a 50-per-week-per-registered-domain ACME cap.

This is the durable half of the mitigation applied during the 2026-09-03 rate-limit incident.

Current state (measured 2026-09-04)

The wildcard: hive-wildcard-tls in ns hive-hub on hive-oke, issuer letsencrypt-dns01 (DNS-01 via Cloudflare). SANs:

*.hive.hivecommons.dev
*.lke.hive.hivecommons.dev
hive.hivecommons.dev

	hive-oke	lke648397
ingress refs to hive-tls-hc	59	136
Certificate objects	23	37
hosts under hive.hivecommons.dev	107	156
hosts NOT wildcard-covered	dibs.kubestellar.io, hive.kubestellar.io	none
--default-ssl-certificate on nginx	set (hive-hub/hive-wildcard-tls)	not set
wildcard secret present	yes	no

So ~195 ingress TLS references and ~60 Certificate objects exist to do what one wildcard already covers.

Why this matters

Per-host certs make the ACME cap structurally reachable: ~263 wildcard-covered hostnames across the two clusters against a 50/week limit. Every hostname change costs a certificate. A live example on hive-hosted-hosted-projectbluefin-knuckle-gjvq — the stored secret still carried the pre-migration SAN

DNS:hosted-projectbluefin-common-nmq5.hive.kubestellar.io
DNS:hosted-projectbluefin-knuckle-gjvq.hive.hivecommons.dev

while the Certificate requested the .hive.hivecommons.dev form of both, so cert-manager reported RequestChanged: Fields on existing CertificateRequest resource not up to date: [spec.dnsNames] and tried to re-issue. That churn is what exhausted the quota.

Confirmed during this investigation: deleting a spoke's hive-tls-hc secret causes cert-manager to mint a brand new per-host certificate rather than falling through to the wildcard — direct evidence the wildcard is not in the serving path.

Work required

1. lke648397 prerequisites (cluster-wide, additive, low risk)

• Copy hive-wildcard-tls from hive-hub on hive-oke into the lke cluster (or issue a second wildcard there with the same DNS-01 issuer - note this costs one ACME issuance).
• Add --default-ssl-certificate=<ns>/hive-wildcard-tls to the lke ingress-nginx-controller args.
• These two alone change nothing observable, since explicit tls: blocks still win. They are just the prerequisite.

2. Provisioner/template change (the actual fix)

• Stop emitting a tls: block on provisioned spoke ingresses (hive, hive-api, hive-contribute, hive-snapshot, hive-terminal) when the cluster domain is wildcard-covered.
• Stop creating the per-hive hive-tls-hc Certificate on those clusters.
• Keep per-host certs only for hosts outside the wildcard: currently dibs.kubestellar.io and hive.kubestellar.io on hive-oke. Note dibs is separately slated to move to dibs.hivecommons.dev (🌱 dibs should be canonically dibs.hivecommons.dev, with dibs.kubestellar.io as a redirect #5925), which is one level up and so is NOT covered by *.hive.hivecommons.dev - it needs its own SAN on the wildcard.

3. Cleanup after the template change lands

• Remove the now-unused hive-tls-hc Certificates and secrets.
• Verify each spoke still serves a valid cert by SNI before deleting anything.

Why this cannot be done by hand

The provisioner reconciles spoke ingresses back within minutes. During the incident, manual issuerRef edits on 15 Certificates were reverted the same way. A kubectl sweep across ~195 objects would not hold - this has to be a template/provisioner change.

Renewal consideration

Once spokes rely on the wildcard, its renewal becomes a single point of failure for every dashboard URL on the domain. Worth adding an alert on hive-wildcard-tls approaching expiry, and keeping the DNS-01 path monitored, since a failed wildcard renewal would take down all ~263 hostnames at once rather than one.

Related: the re-mint loop that consumed the quota is tracked separately; dibs canonical-host move is #5925.

[Danathar]

Item 2 — the provisioner change — is opened as #5981. Refs, not Closes: items 1 and 3 are cluster work.

What it does

wildcard_tls_secret on a cluster in clusters.json makes provisioned spoke Ingresses omit both the per-host tls: block and the cert-manager.io/cluster-issuer annotation.

Both halves, deliberately. The tls: block is what beats --default-ssl-certificate; the annotation is what makes cert-manager's ingress-shim mint the Certificate in the first place. Dropping only the first leaves an orphaned Certificate still churning against the cap; dropping only the second leaves the wildcard inert. The issue asks for both outcomes and they need both edits.

Opt-in per cluster, and why it has to be

I made this off by default rather than deriving it from the domain, because the two states aren't symmetric:

• Wrongly minting a certificate costs one issuance out of fifty.
• Wrongly omitting one makes ingress-nginx serve its built-in self-signed certificate, and every spoke dashboard on the cluster fails TLS validation at once.

Which is precisely the shape of lke648397 today — your table has it as wildcard secret no, --default-ssl-certificate not set. If this shipped as "on wherever the domain looks right", enabling it would have taken lke down rather than saving certificates there. So the flag is an operator assertion that item 1 is done, and the hub does not try to infer it.

Three vetoes, any of which keeps the per-host cert: no flag, OpenShift Route cluster (edge termination, no secret, never used this path), or the host isn't covered.

One thing worth confirming from your side

dibs.hivecommons.dev is a genuine trap here and it's now a test case. It sits one level above *.hive.hivecommons.dev, so the RFC 6125 single-label rule says not covered — matching your note in the issue that it needs its own SAN. A coverage check written on the domain suffix alone would have said "yes, .hivecommons.dev matches" and dropped the tls: block for a host nothing serves.

The same rule means the apex (hive.hivecommons.dev) and any deeper name are excluded too, so coverage is decided per host rather than per cluster and hive.kubestellar.io keeps its own certificate on an opted-in hive-oke without any special-casing.

Tests

Manifest assertions run on the parsed YAML, not substrings — annotations: with an empty body parses as null and would slip past a string check, and hive-api's only annotation is the issuer, so the opt-in has to remove the key itself. Every document is parsed in both modes because the likeliest break in a hand-written YAML template with conditionals is indentation, and that failure lands on a cluster at provision time rather than in CI. Mutation-checked; the terminal's per-hive auth-url is asserted to survive alongside the TLS change.

Not done, flagged rather than guessed

Renewal alerting. You're right that this converts the wildcard into a single point of failure for ~263 hostnames. I didn't implement the alert — it needs a decision about where fleet alerts live, and I'd rather raise that than pick a home for it. It's noted in src/docs/spoke-wildcard-tls.md as a prerequisite for enabling this fleet-wide rather than per-cluster.

Items 1 and 3. No cluster access here. The doc carries the two prerequisite commands and the SNI check to run before deleting anything — worth being deliberate about that one, since your own investigation confirmed that deleting a spoke's secret makes cert-manager mint a fresh per-host cert rather than falling through to the wildcard. So "delete and see" is not a safe probe; check what's being served first.

One consequence of the design worth stating plainly: this changes newly provisioned and re-reconciled spokes. Existing spokes keep their tls: block until the provisioner rewrites their manifest, so item 3's sweep is what actually reclaims the ~60 Certificate objects, and it should follow this rather than precede it.

[clubanderson]

PR #5981 has been reviewed and squash-merged as 2cd730a. It covers item 2 only: provisioned nginx spoke Ingresses now omit the per-host tls: block and cert-manager issuer annotation only when the cluster has opted in with wildcard_tls_secret and the specific host is covered by the single-label wildcard. Hosts outside coverage (dibs.kubestellar.io, hive.kubestellar.io, dibs.hivecommons.dev, apex/deeper/custom domains) still keep per-host TLS; clusters without the opt-in (including current lke648397 state) keep the old behavior.

Remaining manual steps:

1. Finish lke648397 prerequisites before enabling the opt-in there:

kubectl --context hive-oke -n hive-hub get secret hive-wildcard-tls -o json \
  | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.managedFields,.metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"]) | .metadata.namespace="hive-hub"' \
  | kubectl --context lke648397 apply -f -

kubectl --context lke648397 -n ingress-nginx patch deployment ingress-nginx-controller \
  --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--default-ssl-certificate=hive-hub/hive-wildcard-tls"}]'

kubectl --context lke648397 -n ingress-nginx rollout status deployment/ingress-nginx-controller
kubectl --context lke648397 -n ingress-nginx get deployment ingress-nginx-controller \
  -o jsonpath='{.spec.template.spec.containers[*].args[*]}' | tr ' ' '\n' | grep default-ssl-certificate

2. After the deployed hub config sets wildcard_tls_secret only on clusters where the wildcard secret/default cert are confirmed, let/re-run the provisioner reconcile hosted spokes and verify no wildcard-covered spoke Ingress still has a per-host tls: block or issuer annotation:

kubectl --context <ctx> get ingress -A -o json \
  | jq -r '.items[] | select((.metadata.namespace|startswith("hive-hosted-")) and ((.spec.tls // [])|length > 0 or (.metadata.annotations["cert-manager.io/cluster-issuer"] // "") != "")) | [.metadata.namespace,.metadata.name,([.spec.rules[].host] | join(","))] | @tsv'

3. Before deleting any old per-host cert material, verify each spoke serves the wildcard by SNI, then remove only the now-unused per-host Certificates/secrets:

openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject -ext subjectAltName

kubectl --context <ctx> -n <spoke-namespace> delete certificate hive-tls-hc hive-tls --ignore-not-found
kubectl --context <ctx> -n <spoke-namespace> delete secret hive-tls-hc hive-tls --ignore-not-found

Do not close this issue yet: item 1 and the post-reconcile cleanup/renewal monitoring remain operator work.