Support Lazy Loading of Pod Templates

[GWphua]

Description

There may be some circumstances where we want to change the config of the task pod, such as changing the cpu requests/limits of default task pods by editing the base YAML pod template file. In my case, I am editing something that looks like this example ConfigMap, to directly make changes to the pod templates.

However, since the code only updates HashMap<String, PodTemplate> templates during Overlord initialization, the changes are not immediately reflected.

This PR aims to solve this problem and make Druid more adaptable in Kubernetes environments. Instead of internally keeping track of the deserialized PodTemplates we have read from all template files, we will now run the deserialization process whenever a new Task Pod is to be created. Do note this PR does not support reading from newly created pod template files. (e.g. If we create a newPodTemplate.yaml, we will still need to let the Overlord run template initialization for the new file to take effect).

The original intention of fail-fast for invalid pod templates is retained by validating all of the pod templates defined during initialization.

Performance Considerations

Given that a ingestion task will create n task pods, and we have k pod templates defined, we will be reading from the pod template n + k times instead of just k times.

Release note

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[FrankChen021]

I like this feature and it looks good to me