Update controller and agent to kube-rs client 0.91.0

[kate-goldenring]

What this PR does / why we need it:
Our controller uses a really out of date Kubernetes client (kube-rs) API. This updates it to a newer version. Another PR can bump us to latest, which will cause more changes to the agent which is why I avoided that big of a jump.
Much of the controller code was written years ago when the kube-rs did a lot less of the heavy lifting. Switching to a newer version with a better reconciliation model means I was able to delete a lot of helper code.
All the tests had to be rewritten for the new API.

Special notes for your reviewer:
This is a lot. I am hoping the tests can confirm the functionality but we may also need to do a good bug bash.

If applicable:

• this PR has an associated PR with documentation in akri-docs
• this PR contains unit tests
• added code adheres to standard Rust formatting (cargo fmt)
• code builds properly (cargo build)
• code is free of common mistakes (cargo clippy)
• all Akri tests succeed (cargo test)
• inline documentation builds (cargo doc)
• all commits pass the DCO bot check by being signed off -- see the failing DCO check for instructions on how to retroactively sign commits

[diconico07]

Did a first pass here, didn't look at the tests for now.

[diconico07]

Does this play well with Server Side Apply used by the agent ? I fear this would completely mess with fields ownership and prevent agents to remove themselves from an Instance afterwards (on device disappearance).
Also would need to remove the finalizer of the vanished nodes' agent.

[kate-goldenring]

Server side apply is fairly new to me. It sounds like the idea is that only one owner should manage an object or there could be conflicts. The owner of the instance objects should be the agent. Therefore, i should switch this back to do patching as we used to?

[diconico07]

If we remove a node at a time, then I think we should impersonate the node's agent to do so and do what the missing node agent's would do if it were to consider the device disappears.

[kate-goldenring]

I don't like the idea of impersonation, but i don't see removing a finalizer as impersonation. Does d9689ae address your concern

[diconico07]

It would be easier to implement From<OwnershipInfo> for OwnerReference than repeating this block every time

[kate-goldenring]

Thinking about this more, i think we want to avoid using the finalizer API for the instance_watcher since we don't do anything on instance deletes so a finalizer shouldn't be necessary

For node_watcher, we could also remove the finalizer given that we don't care if the node is fully deleted before reconciling it.
Even the pod_watcher is debatable. For all of these, reverting to using a watcher or reflector may be better.

[diconico07]

@kate-goldenring the Controller system doesn't have any "delete" event partly because these can be missed easily, e.g for the node watcher if the controller was on the now missing node, it will never get a delete event for that node.

Overall, the akri controller currently use an event based imperative mechanism (node got deleted, so I do remove it from Instances). But the kube-rs "Controller" system is a bit different in that it works with: I have a resource (in our case the Instance), I should ensure everything is set-up correctly for it.

So if we were to follow the full kube-rs flow here, we would have an Instance controller, that ensure all broker Pods/Jobs/... are here, and that all referenced nodes are healthy, it would trigger on an Instance change to reconcile that Instance, would trigger on owned Pod/Job/... change (to ensure it is always according to spec), and would also trigger (for all Instances) on node state change.

We could also have a reflector for the Configuration to maintain a cache of them and avoid fetching the Configuration from API every time we reconcile an Instance.

I didn't comment on this as your goal was to do the minimal changes necessary to upgrade to latest kube-rs

[kate-goldenring]

@diconico07 that vision makes sense. I started down the path of minimal changes and obviously it grew so much that a full rewrite with just an instance controller would have been easier. At this point, I don't have much more time to devote to this, so could we look at this from the lens of what needs to change here for parity and upgraded kube-rs and then track an issue on rewriting the controller?

[kate-goldenring]

Because we are not reconciling instance deletes anymore, the BROKER_POD_COUNT metric is becoming more of a BROKER_POD_CREATED metric

[kate-goldenring]

I think we need to bump rust version in a separate PR given the failing e2e tests probably due to too low a rust version in the cross containers:

#19 6.424 error: package `opcua-discovery-handler v0.13.2 (/app/discovery-handler-modules/opcua-discovery-handler)` cannot be built because it requires rustc 1.75.0 or newer, while the currently active rustc version is 1.74.1

[kate-goldenring]

it would trigger on an Instance change to reconcile that Instance, would trigger on owned Pod/Job/... change (to ensure it is always according to spec), and would also trigger (for all Instances) on node state change.

@diconico07 clarifying here: are you saying the controller should still watch Instance, Pod and Node resources? or just instance resources and verify the existence of resources based on the instance change?

[diconico07]

It should reconcile Instances (Controller mode), watch Nodes (IIRC there is a way to just get a trigger when a specific field gets updated), and watch Pods,Services,Jobs (if we want to always keep these in line with the spec in Configuration).

The last two being light watches (or probably reflector for nodes), that will just trigger an Instance reconciliation.