[Feature Request] KaaS - CRUD API for Clusters

[schnatterer]

Abstract

For a new offering at Cloudogu we have implemented dynamic provisioning of K8s clusters against the APIs of different cloud providers to make good on our promise of vendor independence.

Implementing this per provider is tedious. I think this is where the SCS KaaS standard could shine!
Couldn't there be a standardized API to do this vendor-independently?
For us, as a consumer, this would provide the real value of a sovereign cloud standard: Implement once, support multiple SCS KaaS-compliant providers.

Initial thoughts

I am new to the SCS community and no expert on the provider side, more like an experienced consumer of Kubernetes clusters.
To get the discussion started, I spent some time analyzing the current situation and came up with some questions that I like to hear your thoughts on.

The biggest challenge I see here is different implementations/APIs among the standardized KaaS providers.
Looking at the existing SCS KaaS-certified products from noris, ScaleUp and Syself, I can see three different implementations: Gardener, ClusterAPI, OpenStack.

From a consumer's point of view, it would be great to be able to choose between different clients like Terraform/OpenTofu, CLI, generic HTTP and (depending on the API) Kubernetes CRDs.

I created a summary of what the different implementations offer in this regard:

Implementation	Terraform/OpenTofu	CLI	HTTP	CRD
OpenStack	openstack	openstack	openstack API	-
CAPI	kubectl/Kubernetes	kubectl	Kubernetes API	Cluster
Gardener	gardener	kubectl / gardenctl	Kubernetes API	Shoot

Can one of these be chosen as the standard KaaS API (like OpenStack in SCS IaaS)?

I can imagine that it will be a challenge to chose one, especially so because they have different attributes (like Kubernetes-native CRDs) and tools involved and would force providers to build adapters. I don't know if this is realistic.

CAPI and Gardener as Kubernetes-native implementations seem closer to each oter than OpenStack.
With a tool like cluster-api-provider-gardener (CAPGa) (unstable, unfortunately)
would we be able to agree on CAPI as the standard?

For OpenStack-based products it would probably be challenging to support CRDs, right?

So maybe the solution here could be that providers either offer an OpenStack API or one Kubernetes-native API?

Creating a new SCS API would cause a lot of effort for all providers and might hinder adoption, so I don't think this would be wise.

What is the most pragmatic solution here?

Stakeholders

In an initial talk with @depressiveRobot I got the impression that he considers this a sensible enhancement for a future version of SCS KaaS.

First, I would like to add the representatives of SCS-compatible KaaS products to the discussion:

• noris Sovereign Cloud (nSC): @benedikt-haug
• ScaleUp Open Cloud: @fzakfeld
• Syself Autopilot: @janiskemper

In addition, I know that Cloud & Heat are considering a KaaS certification, so @horazont @Felkonsky might be interested in this discussion.

At SCS Summit, @majst01 introduced metal-stack.io. I can imagine metal-stack could also bring a lot of experience into this discussion.

Please add all others that might be interested in joining this effort.

Closing thoughts

I'd be willing to join SIG standardization meetings to discuss in person. Just let me know which meeting will be a good fit.

In addition, I can offer to evaluate any POC/MVP of a standardized SCS KaaS CRUD API with different providers and provide feedback and a real-world use-case.

This is my first issue with SCS. Looking forward to working with you all!

[fzakfeld]

Hey @schnatterer,

we had discussions about the definition of "KaaS" before. I think it makes sense to extend the standard to the initial provisioning of clusters and your use case is a good example why that is needed. This of course will create a hurdle for additional KaaS providers looking at getting their offering certified.

Currently, Gardener is the platform most used for certified SCS KaaS (ScaleUp and Noris), while CAPI seems much more "generic" and support with Gardener could work out quite well. We can test plugin together this in our (ScaleUp) staging setup.

I'll add @chrisschwa to the discussion, as they might also be interested in the KaaS certificate and use something other than Gardener or CAPI.

[majst01]

Adding @Gerrit91 to the loop

[janiskemper]

Thanks @schnatterer for this initiative! This has been in discussion for a long time already, since we started working on the SCS KaaS tender.

I personally would extend the picture in the following way: Managing the clusters via API is one thing, but how they work in the background, and which other cloud infrastructure you need, is highly relevant for you.

I'm not the Cloudogu expert, but these are examples of what a software company might look for:

• Storage
• Networking (load balancers, private networks)
• Security configuration (e.g. we at Syself currently don't support Istio ambient mode for reasons)

User stories:

1. You run databases in Kubernetes but a provider offers only replicated storage, which makes this very slow and inefficient
2. You rely on private networking features of the providers and a provider doesn't have it or no good ones (e.g. Hetzner)
3. You have highly specific security configuration that you want in Kubernetes and one provider doesn't support it

There are a lot of other examples, and they can get very specific. The point is that they will be relevant for software companies - it just depends on for how many of them.

I personally think that the effort of having different APIs to manage Kubernetes clusters, as long as they are reasonably similar (e.g. because they work in a declarative way, such as Gardener and Kubernetes), is rather small compared to the other examples.

However, I don't want to discuss about how important a specific example is, I'm more interested in what you think is important to you specifically, apart from the API to manage the clusters @schnatterer .

[schnatterer]

Thanks for your responses.

@fzakfeld ah, so no OpenStack Magnum? I must have gotten this wrong. It would have been an obvious choice coming from SCS IaaS. Would that API still be a valid candidate for SCS KaaS even if no one certified with it so far?

@janiskemper yes you're right, mostly Kubernetes clusters run in the context of other cloud resources. I kept that out of my proposal for two reasons:

• keep the problem as simple as possible to get results more quickly
• our use case is really that simple at the moment. We only want to create and delete clusters. In the long run we will also need S3-compatible storage for backups, but my first thought was to get it via SCS IaaS. At the moment we are hoping to get feedback from the market as soon as possible. Maybe this feedback will then tell us that the databases are slow with provider xyz or that they need to integrate the cluster into their private networks, etc, like you said.
  What would your idea be to prepare for these issues but not make the problem field so wide that we set out into long discussions?

Having another look at the status quo of the three KaaS-certified offerings I can already see an implicit advantage of SCS KaaS: All are relying on generic open source APIs instead of proprietary ones. So at Cloudogu we could already integrate a generic Gardener or CAPI provider, even without SCS. The downside for SCS is that the term SCS is not (yet) showing there.

[janiskemper]

I see the point to move faster with reducing the scope. I'm skeptical, however, if this is just a small feature.

The currently most logical and easy way would be, as already mentioned before, to use the CAPI API with its topology feature. That's what we have currently. And that's ironically something we'll move away from in the future because it is extremely ugly for users.

The standardized API won't have any effect if it is not part of the standards. If we make it part of the standards, suddenly any provider would be forced to implement this on top of their own API. That's significant work.

At the same time, I didn't entirely understand yet your exact use case and how this standardized API really makes a lot of difference.

The CAPI cluster object contains a lot of custom values that you can inject. These custom values will be per provider. Therefore, it will not be possible anyway to create a "well-configured" (whatever that means) cluster on different providers with the exact same configuration. You would still have to vary it to some degree.

However, as soon as you already need a slightly different configuration, it is no work to also do a bit more changes.

We at Syself are actually at a similar position as you are, because we want to build on top of many infrastructure providers with our KaaS solution. Therefore, I have already thought about such things a lot ;)

One significant aspect for you would also just be that the API works in a stable and easy way. So that you get direct feedback when a cluster is ready to use, etc. Obviously, standardizing this process or the keys and values in the API does make it a bit easier for you.

I'm still skeptical though that, given already the fact that SCS assumes to have such a stable API because it is necessary also for the SCS test suite, that cost (on KaaS Provider side) and benefits of the standardized API really make it worth it.

[fzakfeld]

The standardized API won't have any effect if it is not part of the standards

That's a good point. For us, we expose Gardener with all it's tooling / conventions (Shoot resources, gardenctl, dashboard) to customers and built our processes (project provisioning, quotas, billing, ...) around it. Offering CAPI via extension would always be an extra, not something we'd recommend by default. But being a special case would go against SCS.

@flyersa has experience with Magnum, adding him as well

[schnatterer]

@janiskemper

on CAPI:

And that's ironically something we'll move away from in the future because it is extremely ugly for users.

That is an important insight that sounds rather disqualifiying for CAPI.
Can you already speak about what your plans are?

The CAPI cluster object contains a lot of custom values that you can inject. These custom values will be per provider.

Would Gardener as an API handle this differently?
Or would it always mean for SCS to standardize the keys and values inside an API like CAPI/Gardener?

I'm still skeptical though

I think it is a fair to allow for a possible outcome of this story to decide that standardizing the API is too expensive from a provider site.
Would you're preference be to leave the standard as is (everyone can choose their API freely)? Or to name a few APIs that are allowed for standardized SCS like Gardener and ClusterStacks/CAPI?
For example, looking ahead, if new providers with their own maybe even proprietary API would like to be certified, would their new API just be added to the conformance tests or would we want to limit it to a chosen few well-established APIs?

I didn't entirely understand yet your exact use case and how this standardized API really makes a lot of difference

Somewhat difficult to find the right balance of not writing too much. I'll just add some more context and technical detail and you ignore it if it is too much info :)

Cloudogu offers the ecosystem, an operational platform for operating sovereign tool-stacks.
For example, you can choose from multiple project management tools (e.g. OpenProject, Redmine, etc.) and wiki tools and deploy them on an any kubernetes cluster you want.
It is important to note that we provide the platform, not the tools or infra. Here is a simple (marketing 😅) diagram that I think transports the general idea

In the past our platforms made it easy to deploy and operate the tools and the platform.
Now we want to make the whole process easier by also including the infra (let's say the cluster) in one click (or commit).

As I said, we have built that for different providers. To me it seems that the difference in terms of parameters (from a user point of view) are few. What we need is

• number of (initial) nodes
• node type
• location
• authentication

We built this using the individual TF providers. We will likely go on adding other providers in the future. But this effort sparked the idea that it would be great to provide an option SCS-compatible where only an additional property for the endpoint of the provider would be needed and everything else would be standardized.

After you wrote that the SCS test suite already knows about a stable API, I spent some time today getting a rudimentary understanding of how these automated compliance tests work. On first look, it seems to me that the gardener setup with noris and scaleup could come close to my idea of just a different endpoint:

standards/playbooks/clusters.yaml.j2

Line 109 in b5a1727

garden_api_server: "{{ clouds_conf.scaleup_api_server }}"

But if agreeing on Gardener as the standard API makes sense from a compatibility, adoption, effort, political, etc. point of view I cannot say.

[janiskemper]

This makes a lot of sense, @schnatterer!

CAPI's cluster object, with ClusterClass, works in a way that you have key value pairs where you can inject your custom configuration. So you have a standard object with fixed values, and then you have an "open" section where you can do anything.

Important is that it does NOT standardize actually node types, location format, or authentication methods.

The node types will depend fully on the provider (if I am not totally wrong the SCS IaaS layer standardizes them). The locations as well. The authentication method also and that will not even have a fixed API, because e.g. in our own Cluster object you can inject your IDM for OIDC access to the cluster, which obviously works differently elsewhere.

So a standardized API where you have a fixed value to decide how many control planes and worker nodes you wanna have is actually "easy" but also probably not really what creates the work for you.

I think for you it will be more relevant to have

• knowledge about the server types ( to choose the right size)
• knowledge about how authentication works (and how to automate it)
• etc

What we wanna do at Syself is to replace this very generic and ugly part of Cluster API's object, where you can inject any value, to a more structured and validated object that we can construct in a way that it is user-friendly.

As long as we decide to fully control an SCS KaaS API, we'll always have to take an approach similar to CAPIs, with mostly generic parts and then some variable ones. That will never give you a great user experience though.

I think this discussion is very interesting and I'd like to see how we can connect both UX of the API, and something that facilitates your life at Cloudogu!

[benedikt-haug]

I've recently extended the noris Sovereign Cloud (nSC) user docs how to spawn gardener clusters via API/kubectl: https://noris.cloud/en/nsc/gardener-kubernetes/#creating-clusters-via-api

The relationship between gardener and CAPI from the gardener perspective can be found here: https://gardener.cloud/docs/gardener/concepts/cluster-api/

TLDR;
I see little reason to establish a gardener CAPI provider. It would likely be just be a worse replacement for the excellent status quo.

[berendt]

I think service discovery would be a good solution to this problem. Imagine a file at https://<provider>/.well-known/managed-kubernetes.json with the following content. Of course, this is just a rough example meant to illustrate the idea.

{
  "schema_version": "v1",
  "conformance": ["https://docs.scs.community/standards/scs-kaas-v1"],

  "provisioning": {
    "interface": "gardener",
    "endpoint": "https://api.garden.example.com"
  },

  "regions": [
    { "name": "de-north-1", "country": "DE", "jurisdiction": "EU" }
  ],

  "node_types": {
    "naming_scheme": "https://docs.scs.community/standards/scs-0100-v3",
    "available": [
      { "name": "SCS-2V-4-20", "vcpus": 2, "ram_gb": 4, "disk_gb": 20 }
    ]
  },

  "storage_classes": [
    { "name": "block-default", "default": true, "replicated": true }
  ],

  "not_standardized": ["ingress_controller", "cni_choice"]
}

[schnatterer]

Let me try to summarize where this discussion got as so far.

• For providers using Gardener, a CAPI frontend/adapter would not make sense and the other way round, right @fzakfeld @janiskemper @benedikt-haug ?
  A sensible trade-off might be to allow a number of generic well known APIs like Gardener, CAPI and maybe OpenStack Magnum. From my perspective this would still be much better than every provider bringing their own (possibly proprietary) APIs. For example, what I am missing at the moment is a "Getting started" document in SCS User docs. With only two or three providers it would be doable to describe how to get started with these and different default tools like TF or CLI tools. Or do you think the effort would outweigh the benefit?
• A service discovery endpoint as proposed by @berendt could solve the problem of provider specific parameters like regions and node types. I think this could work for our use case at Cloudogu: We could query the end point, use interface to decide which implementation to use and then populate the fields using the values from the service discovery endpoint. The service discovery endpoint could also help humans. We could create a standardized fact sheet of different providers on the SCS page so you can quickly see what's on offer. This would work well with the getting started docs I mentioned above, too.

I'd also be very much interested to hear the opinions of providers not yet standardized. Do you see value for the standard in this idea? Would this cause a lot more effort for your certification process or even scare you off?
@horazont @Felkonsky
@majst01 @Gerrit91
@flyersa

[flyersa]

We are an early adopter of using magnum with external capi drivers, mainly using the vexxhost one. This is by now in Openstack also the only way todo magnum.

By now this is stable and works great, and even have features now that tenant cluster controlplanes do not need to be public available by adding them to the metadata network. And they work with the usual capi images.

It does not provide the clusterapi or gardener functionality of shared controlplane or multi provider clusters, but what it does great is the integration into the openstack ecosystem. But i dont see why multi provider k8s should be a requirememt with multiple own dcs and availablity zones. How often did people at all migrate between scs clouds so far? (Which is a point why we have this standards)

We use it because we do not need to maintain a different system and can use native openstack apis to spool up clusters. We also did alot of work to make this work completly airgapped. The vexxhost driver is also very simple and not a big software blob like gardener and easy to contribute to.

So imho yes this should be in the scope for scs kaas especially because it is a native openstack ecosystem. We maintain around 60 clusters this way atm without any issues, even with custom build pipelines to build our own images or customers doing that themself for additional features.

Maybe a gardener driver for magnum would be a idea too (but may be stupid no idea). For the pure other part i like christians idea too.

[depressiveRobot]

This is a great discussion so far with lots of ideas we can build on.

@schnatterer Thank you very much for taking the initiative.

I will put this topic on the agenda for the next SIG Standardization/Certification meeting on 2026-08-20 at 2pm. I would be happy if as many of you as possible could join us.

Let me add a few more people here, who are to my knowledge also working on KaaS. Feel free to add your thoughts on this too 🙂
@mbuechse, @cwrau, @mpilka