feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks #14642
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kontinuation
changed the title
Kontinuation
changed the title
Kontinuation
changed the title
|
|
alamb
approved these changes
Feb 13, 2025
2010YOUY01
approved these changes
Feb 14, 2025
|
Thanks again @Kontinuation and @2010YOUY01 |
|
This PR is merged but for some reason the github ui is not showing it: |
jonahgao
pushed a commit
to jonahgao/datafusion
that referenced
this pull request
Feb 14, 2025
…ultiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
--memory-limitfor all benchmarking tools #14641.Rationale for this change
I had to run sort-tpch queries with memory limit when testing fixes for memory related issues, so I decide to add
--memory-limitoption for most of the benchmarking cli tools. I wish other developers could find it handy.What changes are included in this PR?
This PR adds 2 cli options
--memory-limit,--mem-pool-typeand--sort-spill-reservation-bytesto the following benchmarking tools:dfbenchsubcommands:sort,sort-tpch,clickbench,h2o,imdb,parquet-filtertpchimdbexternal_aggralready supports--memory-limit, it now accepts--mem-pool-type. The default value of--mem-pool-typeisfairso the behavior remains unchanged.Are these changes tested?
The changes were tested manually.
Are there any user-facing changes?
No. The benchmarking guide has not covered every option so hopefully the developers could find these options themselves using
--help.