Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ghalistener high cardinality metrics #3670

Open
4 tasks done
christophermichaeljohnston opened this issue Jul 19, 2024 · 4 comments · May be fixed by #3671
Open
4 tasks done

ghalistener high cardinality metrics #3670

christophermichaeljohnston opened this issue Jul 19, 2024 · 4 comments · May be fixed by #3671
Labels
bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode needs triage Requires review from the maintainers

Comments

@christophermichaeljohnston
Copy link

Checks

Controller Version

0.9.2

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

All actions scheduled by ghalistener use a new runner causing a new metric for every single action. This is because the metrics include runner_id and runner_name which is distinct for every run. For example:

gha_completed_jobs_total{<snip>,runner_id="71363",runner_name="self-hosted-linux-x64-zfhfn-runner-k752n"} 1
gha_completed_jobs_total{<snip>,runner_id="71369",runner_name="self-hosted-linux-x64-zfhfn-runner-pr56c"} 1
gha_completed_jobs_total{<snip>,runner_id="71376",runner_name="self-hosted-linux-x64-zfhfn-runner-qns9x"} 1

The <snip> labels above are identical for the same workflow, but there is a new metric for each action due to runner_id and runner_name being unique.

This also causes memory and cpu usage to continually creep as the listener must keep track of all these metrics, even though it will never update them, due to the unique labels.

Describe the bug

^ see above

This was fixed in githubrunnerscalesetlistener in #3003 and the fix needs to be included in ghalistener.

Describe the expected behavior

Metrics should not include labels are unique as this causes high cardinality and renders the counters, which will only have a value of 1, as unusuable.

Additional Context

n/a

Controller Logs

n/a

Runner Pod Logs

n/a
@christophermichaeljohnston christophermichaeljohnston added bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode needs triage Requires review from the maintainers labels Jul 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

@christophermichaeljohnston christophermichaeljohnston linked a pull request Jul 19, 2024 that will close this issue
Open
@christophermichaeljohnston
Copy link
Author

Screenshot 2024-07-19 at 9 14 02 AM

^^ gha listener memory and cpu usage increase caused by tracking of high cardinality metrics

@iwaffles
Copy link

Related to #3153

@ajschmidt8 ajschmidt8 mentioned this issue Sep 18, 2024
Open
@christophermichaeljohnston
Copy link
Author

Included in the attached PR for removal is job_workflow_ref which also causes horrible high cardinality

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working gha-runner-scale-set Related to the gha-runner-scale-set mode needs triage Requires review from the maintainers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

metrics cardinality for ghalistener christophermichaeljohnston/actions-runner-controller
2 participants
@iwaffles @christophermichaeljohnston