Introduce node level circuit breaker settings for k-NN

[markwu-sde]

Description

KNN plugin currently uses a cluster-wide circuit breaker. This doesn't work as well as it could when nodes have different memory capacities.

Solution

Added node-specific circuit breaker limits using node attributes. This opens up flexibility for heterogenous circuit breaker limits.

Usage

1. Set node attribute in opensearch.yml:

node.attr.knn_cb_tier: "integ"

2. Configure limit for that tier:

PUT /_cluster/settings
{
  "persistent": {
    "knn.memory.circuit_breaker.limit.integ": "75%"
  }
}

Implementation

• Uses OpenSearch's groupSetting for dynamic limit configurations
• Cache initially uses cluster-level default
• Updates to node-specific limit when node attributes are available
• Falls back to cluster default if no node-specific limit exists

Testing

Modified KNNCircuitBreakerIT integration tests to include node-level CB.

Used OSB benchmarking to run a modified low load test with/without the node level circuit breaker.

• Added node.attr.knn_cb_tier = 'integ' to build.gradle
• Ran 5-query search-only vector benchmark after indexing
• Verified memory usage and circuit breaker behavior

Initial state (no limits set):

{
  "graph_memory_usage_percentage": 1.35,
  "graph_memory_usage": 110056
}

After setting node limit to 500000kb:

curl -XPUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent" : {
    "knn.memory.circuit_breaker.limit.integ" : "500000kb"
  }
}
'

Results:

{
  "graph_memory_usage_percentage": 22.01,
  "graph_memory_usage": 110056,
  "circuit_breaker_triggered": false
}

Setting cluster limit to 5kb didn't affect node with specific limit which confirms proper override behavior.

Related Issues

Resolves #2263

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[markwu-sde]

Running into an issue very similar to #2223 for the build failures

java.lang.AssertionError: should not be called by a cluster state applier. reason [the applied cluster state is not yet available]
»       at org.opensearch.cluster.service.ClusterApplierService.assertNotCalledFromClusterStateApplier(ClusterApplierService.java:443)
»       at org.opensearch.cluster.service.ClusterApplierService.state(ClusterApplierService.java:229)
»       at org.opensearch.cluster.service.ClusterService.state(ClusterService.java:182)
»       at org.opensearch.cluster.service.ClusterService.localNode(ClusterService.java:166)
»       at org.opensearch.knn.index.KNNSettings.getKnnCircuitBreakerLimitForNode(KNNSettings.java:652)

Since node attributes are not dynamically configurable currently we can cache the value on node initialization when we refresh the cache. This would prevent the update thread from trying to access the attributes via the cluster state.

[markwu-sde]

Node attributes aren't available during the time the cache manager computes its size on initialization. With the additional dependency on node attributes to the circuit breaker we may need to check if the circuit breaker needs to be updated after the node has bootstrapped.

I only see 1 example of a listener being attached in here but attached the listener on KnnCircuitBreaker initialization due needing to refresh the cache to recompute the size and caching the attribute value. If there's any feedback on this that'd be appreciated.