Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rls: Fix flaky test Test/ControlChannelConnectivityStateMonitoring #8055

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

eshitachandwani
Copy link
Member

fixes #5468
The test flakes because

    • Test creates a ClientConn with service config setting the LB policy to "rls"
    • RLS LB policy initializes the control channel to the RLS server. As part of this, it spawns a goroutine to monitor the control channel connectivity state changes.
    • But by the time the first RPC is successfully made, the above goroutine has not gotten a chance to run yet.
    • And at this time, the test stops the RLS server. This moves the control channel to IDLE and it is only now that the monitoring goroutine gets to run, and it has already missed the first transition to READY.

FIX : Use channel to make sure the go routine starts

  1. Our current state change API is lossy because state changes can be lost between the former returning and the caller invoking GetState
    FIX : The fix is to use grpcsync.pubsub to subscribe to the state changes so that we do not loose state changes.

RELEASE NOTES: N/A

Copy link

codecov bot commented Jan 30, 2025

Codecov Report

Attention: Patch coverage is 72.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 82.31%. Comparing base (be12ee9) to head (fa6d028).
Report is 21 commits behind head on master.

Files with missing lines Patch % Lines
balancer/rls/control_channel.go 72.00% 5 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8055      +/-   ##
==========================================
+ Coverage   82.03%   82.31%   +0.27%     
==========================================
  Files         384      386       +2     
  Lines       38750    39056     +306     
==========================================
+ Hits        31790    32147     +357     
+ Misses       5632     5578      -54     
- Partials     1328     1331       +3     
Files with missing lines Coverage Δ
balancer/rls/control_channel.go 89.26% <72.00%> (-3.87%) ⬇️

... and 60 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky test: 1/10k: ControlChannelConnectivityStateMonitoring
3 participants