Support partial scale-down in RayJob

[eric-higgins-ai]

What would you like to be added:
Currently the RayJob admission webhook returns an error when the job sets enableInTreeAutoscaling: true. It makes sense that autoscaling up isn't supported, but I would expect autoscaling down could be supported via the "dynamic reclaim" feature.

I'd potentially be down to implement this - I just wanted to open an issue to see how people feel about this feature before spending time on it.

Why is this needed:
We want to run hyperparameter sweeps with Ray Tune, and they have a feature that allows early exiting from trials based on metric values. With this, it's possible that at some point the job doesn't need all the resources it was initially given (because most trials have finished), and we'd like to be able to reclaim those resources.

Completion requirements:
RayJobs support dynamic reclaiming if the Ray autoscaler indicates the job should scale down

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

It makes sense that autoscaling up isn't supported, but I would expect autoscaling down could be supported via the "dynamic reclaim" feature.

Potentially, we also have a design (not implemented) for generic (any CRD) dynamic Jobs in #77. Maybe it is time to prioritize that work.

cc @mwielgus @mwysokin @tenzen-y

[tenzen-y]

It makes sense that autoscaling up isn't supported, but I would expect autoscaling down could be supported via the "dynamic reclaim" feature.

Potentially, we also have a design (not implemented) for generic (any CRD) dynamic Jobs in #77. Maybe it is time to prioritize that work.

cc @mwielgus @mwysokin @tenzen-y

Yeah, I think so too. Could we prioritize this after the next minor release (0.12)? Because we have a lot of alpha features and trying to fix bugs in this release cycle. This seems to be a slightly big feature.