feat(k8s): watch specific namespaces (PoC)

[lukidzi]

Motivation

By default, Kuma watches for changes across all namespaces. This simplifies the initial user experience, making it easier to get started. However, at some point, users may want to restrict the information available to the control plane to enhance security, or reduce unnecessary resource consumption.

Implementation information

This feature requires modifying permissions across multiple components:

1. Extracting ClusterRole options into Role
2. Restricting webhooks to operate only within specific namespaces
3. Kubernetes manager
4. Event listener

If watchNamespaces is set:

Step 1:

I moved most namespace-scoped resources to separate Role objects, which are created in watchNamespaces and must be set up by the user before deployment. Instead of using only a ClusterRole, we now create a ClusterRole with a more limited scope, a Role in the system namespace, and Roles in each user-provided namespace.

Step 2:

In Helm, we validate whether the user has provided specific namespaces. If so, we configure webhooks to only monitor those namespaces along with the systemNamespace.

Step 3:

We need to define a defaultNamespaces parameter for the Kubernetes cache. This cache retrieves information from the Kubernetes API and, by default, fetches resources from all namespaces. This change introduces a way to limit requests to specific namespaces: watchNamespaces, systemNamespace, and the CNI namespace.

Additionally, in many parts of the code, we perform .List operations, which require a namespace to be specified. Instead of adding the namespace to each request individually, I introduced a wrapper that iterates over and requests resources from each watched namespace.

Since some resources are cluster-scoped, I also introduced a way to check whether a resource is cluster-scoped. If it is, the request is made for all resources. Because defaultNamespaces configures the default informers in the manager, we don't need to define predicates for each controller.

Step 4:

We create separate informers using the Kubernetes client, which fetches resources differently from the default cache. Since there is no built-in option to list resources across multiple namespaces, we need to create an informer for each namespace.

Supporting documentation

https://medium.com/@timebertt/kubernetes-controllers-at-scale-clients-caches-conflicts-patches-explained-aa0f7a8b4332

xref: https://github.com/Kong/kong-mesh/issues/7408

[github-actions]

Reviewer Checklist

🔍 Each of these sections need to be checked by the reviewer of the PR 🔍:
If something doesn't apply please check the box and add a justification if the reason is non obvious.

• Is the PR title satisfactory? Is this part of a larger feature and should be grouped using > Changelog?
• PR description is clear and complete. It Links to relevant issue as well as docs and UI issues
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as an image registry)
• IPv6 is taken into account (.e.g: no string concatenation of host port)
• Tests (Unit test, E2E tests, manual test on universal and k8s)
  • Don't forget ci/ labels to run additional/fewer tests
• Does this contain a change that needs to be notified to users? In this case, UPGRADE.md should be updated.
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label)

[lukidzi]

Closing, let's reopen when it's required