[META] Tiered Caching 3.0 changes

[peteralfonsi]

Please describe the end goal of this project

Tiered caching is mostly complete in versions 2.17 onwards. However there are some small non-backwards-compatible changes we can make when 3.0 comes out.

New features

• Track the fraction of incoming requests that are cacheable at the IndicesService level and expose this, likely in the nodes stats API. This will help users determine whether changing the new indices.requests.cache.maximum_cacheable_size might be helpful to them, and see the effects after turning it on.
• Track the average caching latency and expose this, likely in the cache-implementation-level stats API

Features to remove

• Remove FeatureFlags.PLUGGABLE_CACHE
• Remove the deprecated indices.requests.cache.size setting in favor of cache implementation-specific settings like indices.requests.cache.opensearch_onheap.size, indices.requests.cache.tiered_spillover.disk.store.size, etc
• Remove the old indices request cache stats tracked through ShardRequestCache in favor of the cache-level stats implementation
• Remove documentation about how TC is an experimental feature

Supporting References

These are fairly small changes so we don't have RFCs around them. Overall TC issue is here

Issues

• Remove feature flag: [Feature Request] [Tiered Caching] Remove feature flag for tiered caching #17343
• New stats: [Feature Request] [Tiered Caching 3.0] Additional IRC stats #17366

Related component

Search:Performance