Questions about benchmark

[flyingcat]

Some days ago, I wrote a simple benchmark tool. Nothing new but a copycat of benchcat in this project and alacritty/vtebench. What curious me was that the test results were unstable. Slow running items may vary ±20%, and may influence the next item.

• What does this way of benchmark measure? It seems WriteFile will just return if some internal state is not blocking, so accordingly downstream reading can't be measured accurately.
• Where does this instability come from? Limitations in this way or I do something wrong? Maybe some warm up or larger payload may make it better.

Uh, there is no Readme. If anyone is interested in running it:

scoop install main/luajit
luajit .\bench.lua

[lhecker]

I'm getting the following results on my first run in Windows Terminal Canary 1.23.2913.0:

terminal size 120x30
cat-a                      131.060 / 0.409            320.361 MB/s
cat-u                      131.796 / 1.416             93.105 MB/s
cat-g                      132.744 / 1.791             74.116 MB/s
ccat-a                     131.551 / 2.477             53.100 MB/s
ccat-u                     129.056 / 5.558             23.220 MB/s
bat-a                      128.580 / 2.613             49.203 MB/s
bat-u                      129.583 / 2.872             45.127 MB/s
cursor-a                   128.826 / 3.376             38.158 MB/s
cursor-u                   128.021 / 3.201             39.998 MB/s
cell                       130.151 / 2.390             54.457 MB/s
edit                       128.577 / 2.008             64.016 MB/s

On my second run:

terminal size 120x30
cat-a                      131.060 / 0.415            316.047 MB/s
cat-u                      131.796 / 1.368             96.358 MB/s
cat-g                      132.744 / 1.705             77.866 MB/s
ccat-a                     131.551 / 2.451             53.675 MB/s
ccat-u                     129.056 / 5.411             23.851 MB/s
bat-a                      128.580 / 2.705             47.528 MB/s
bat-u                      129.583 / 2.826             45.856 MB/s
cursor-a                   128.826 / 3.566             36.129 MB/s
cursor-u                   128.021 / 3.261             39.252 MB/s
cell                       130.151 / 2.369             54.930 MB/s
edit                       128.577 / 2.030             63.353 MB/s

It seems the results are stable for me and there's no obvious fluctuation.

It seems WriteFile will just return if some internal state is not blocking, so accordingly downstream reading can't be measured accurately.

WriteFile on a console handle on Windows is fully 100% synchronous. It will only return once the console server (conhost) has fully processed the message. That's also why very tiny writes on Windows are so abysmally slow: The write from your process has to go through the kernel into another process, which then processes it, has to transition back again through the kernel, and ready your process again. This results in severe cross-process chatter and is affected by the relatively high scheduling latency. I was thinking of adding buffering to the console pipe in the future to make it async, but it will be a difficult change to make.

Where does this instability come from? Limitations in this way or I do something wrong? Maybe some warm up or larger payload may make it better.

You mentioned yesterday that your PC is very old. Is it possible that it's thermally throttling? Or are you benchmarking conhost instead of Windows Terminal? Older versions of conhost had various issues with scrolling and had worse lock contention.

BTW: Your benchmark code is really cool!

[flyingcat]

You mentioned yesterday that your PC is very old. Is it possible that it's thermally throttling? Or are you benchmarking conhost instead of Windows Terminal? Older versions of conhost had various issues with scrolling and had worse lock contention.

It's a cool enough desktop PC. The benchmark is for compiled WindowsTerminal.exe of main branch and my PR branch. Something like this, mine vs. main:

It's not that frequently, though. Likely happens after booting, awakening, compiling, or some long time. I need to exclude slow items(cat-u, cat-g) to get more stable results. This will be more noticeable if I maximize the window and make it slower. Maybe related with my system running environment? I'll investigate further when I have time.

WriteFile on a console handle on Windows is fully 100% synchronous. It will only return once the console server (conhost) has fully processed the message. That's also why very tiny writes on Windows are so abysmally slow: The gwrite from your process has to go through the kernel into another process, which then processes it, has to transition back again through the kernel, and ready your process again. This results in severe cross-process chatter and is affected by the relatively high scheduling latency. I was thinking of adding buffering to the console pipe in the future to make it async, but it will be a difficult change to make.

I only understand half of it. I will try my best to understand all tomorrow, going to bed now. It's always a pleasure to receive your reply.