Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce appearance tests in Web Compat to 10 tests picked at random #7272

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

foolip
Copy link
Member

@foolip foolip commented Feb 13, 2025

These were picked with the following command:

cat original-list-of-tests | sort -R | head -n10 | sort

These were picked with the following command:

> cat original-list-of-tests | sort -R | head -n10 | sort
@foolip foolip added the do not merge yet Disable auto-merge label Feb 13, 2025
@gsnedders
Copy link
Member

gsnedders commented Feb 13, 2025

This would cause us to end up with these results.

That results in, for compute-kind-widget-generated specifically:

Old New Diff
Chrome 100% 100% 0%
Edge 100% 100% 0%
Firefox 75.2% 60% -15.2%
Safari 89.3% 80% -9.3%

[Edit: hah, beaten by @foolip by virtue of trying to write a table summarising the change!]

Copy link
Contributor

@jgraham jgraham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine as a placeholder change, but we should work out a way to get more coverage in a reasonable number of tests.

@gsnedders
Copy link
Member

There's 45 other tests, so this takes us from 94.6% of the Web Compat focus area to 18.2%.

Given the current scores of the whole directory, any subset of 10 tests here will alter the Firefox score for the directory by at least 5%. We could go up to 20 tests, which would be 30.8% of the focus area, and be able to better approximate the current scores. That said, I'm not trying to assert we change the current plan to move to 10 tests.

Let's cross-reference this to web-platform-tests/rfcs#167 given subtests might help. It's also potentially related to broader questions about balancing scoring across features within a focus area for Interop, and this is not the first time this has come up.

@gsnedders
Copy link
Member

FWIW, per-element:

Element Chrome Edge Firefox Safari
select-dropdown-box 100.0% 100.0% 22.7% 63.6%
select-menulist-button 100.0% 100.0% 22.7% 100.0%
progress 100.0% 100.0% 59.1% 100.0%
button 100.0% 100.0% 77.3% 72.7%
input-button 100.0% 100.0% 77.3% 72.7%
input-reset 100.0% 100.0% 77.3% 72.7%
input-submit 100.0% 100.0% 77.3% 72.7%
color-input 100.0% 100.0% 77.3% 72.7%
input-search 100.0% 100.0% 77.3% 81.8%
meter 100.0% 100.0% 77.3% 100.0%
textarea 100.0% 100.0% 77.3% 100.0%
input-text 100.0% 100.0% 77.3% 100.0%
select-listbox 100.0% 100.0% 77.3% 100.0%
input-search-text 100.0% 100.0% 77.3% 100.0%
link 100.0% 100.0% 100.0% 100.0%
range 100.0% 100.0% 100.0% 100.0%
checkbox-input 100.0% 100.0% 100.0% 100.0%
radio-input 100.0% 100.0% 100.0% 100.0%

And per-property:

Property Chrome Edge Firefox Safari
background-attachment 100.0% 100.0% 33.3% 100.0%
background-clip 100.0% 100.0% 33.3% 100.0%
background-origin 100.0% 100.0% 33.3% 100.0%
background-position 100.0% 100.0% 33.3% 100.0%
background-size 100.0% 100.0% 33.3% 100.0%
border-image-outset 100.0% 100.0% 33.3% 100.0%
border-image-repeat 100.0% 100.0% 33.3% 100.0%
border-image-slice 100.0% 100.0% 33.3% 100.0%
border-image-source 100.0% 100.0% 33.3% 100.0%
border-image-width 100.0% 100.0% 33.3% 100.0%
border-block-end-width 100.0% 100.0% 83.3% 66.7%
border-block-start-width 100.0% 100.0% 83.3% 66.7%
border-bottom-width 100.0% 100.0% 83.3% 66.7%
border-top-width 100.0% 100.0% 83.3% 66.7%
border-block-end-color 100.0% 100.0% 88.9% 66.7%
border-block-end-style 100.0% 100.0% 88.9% 66.7%
border-block-start-color 100.0% 100.0% 88.9% 66.7%
border-block-start-style 100.0% 100.0% 88.9% 66.7%
border-bottom-color 100.0% 100.0% 88.9% 66.7%
border-bottom-style 100.0% 100.0% 88.9% 66.7%
border-top-color 100.0% 100.0% 88.9% 66.7%
border-top-style 100.0% 100.0% 88.9% 66.7%
border-inline-end-width 100.0% 100.0% 83.3% 94.4%
border-inline-start-width 100.0% 100.0% 83.3% 94.4%
border-left-width 100.0% 100.0% 83.3% 94.4%
border-right-width 100.0% 100.0% 83.3% 94.4%
border-inline-end-color 100.0% 100.0% 88.9% 94.4%
border-inline-end-style 100.0% 100.0% 88.9% 94.4%
border-inline-start-color 100.0% 100.0% 88.9% 94.4%
border-inline-start-style 100.0% 100.0% 88.9% 94.4%
border-left-color 100.0% 100.0% 88.9% 94.4%
border-left-style 100.0% 100.0% 88.9% 94.4%
border-right-color 100.0% 100.0% 88.9% 94.4%
border-right-style 100.0% 100.0% 88.9% 94.4%
background-color 100.0% 100.0% 88.9% 100.0%
background-image 100.0% 100.0% 88.9% 100.0%
border-bottom-left-radius 100.0% 100.0% 88.9% 100.0%
border-bottom-right-radius 100.0% 100.0% 88.9% 100.0%
border-end-end-radius 100.0% 100.0% 88.9% 100.0%
border-end-start-radius 100.0% 100.0% 88.9% 100.0%
border-start-end-radius 100.0% 100.0% 88.9% 100.0%
border-start-start-radius 100.0% 100.0% 88.9% 100.0%
border-top-left-radius 100.0% 100.0% 88.9% 100.0%
border-top-right-radius 100.0% 100.0% 88.9% 100.0%

This doesn't make it super obvious what the one test for Safari should be, nor what the two/three tests for Firefox should be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not merge yet Disable auto-merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants