-
-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(consumers): rust consumers quantized rebalance #6561
base: master
Are you sure you want to change the base?
Conversation
Ok(delay_secs) => match delay_secs { | ||
Some(secs) => match secs.parse() { | ||
Ok(v) => Some(v), | ||
Err(_) => None, | ||
}, | ||
None => None, | ||
}, | ||
Err(_) => None, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit:
I had a little difficulty understanding this, it might be a little more easier to read if we do
let Ok(delay_secs) = runtime_config::get_str_config(...) else {
return None;
}
delay_secs.parse().ok()
Or some variation to reduce nesting.
But this is on the aesthetic side of things
rebalancing::delay_kafka_rebalance(secs); | ||
handle.signal_shutdown(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am I correct in assuming that this closure that is passed in runs in a separate thread, so the main loop is not blocked while it is sleeping before signaling shutdown?
This seems reasonable to me, do we plan to have quantized rebalancing on the other end? I.e. Making the consumer wait a little bit before joining the consumer group |
Add the ability to do quantized rebalancing for consumers, toggleable by runtime config.
What is quantized rebalancing?
Rebalancing events are synchronized to the tick of the clock. Controlled by the runtime config:
quantized_rebalance_consumer_group_delay_secs__{consumer_group}
Let's say:
SET quantized_rebalance_consumer_group_delay_secs__spans=15
timestamp % 15 == 0
timestamp % 15 == 0
What else is in this PR?
This PR adds the ability to read from redis config directly. This was mostly done because I couldn't get the python bindings to work on my machine, it will still fall back to the old implementation if there is an error in the direct implementation