-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce appearance tests in Web Compat to 10 tests picked at random #7272
base: master
Are you sure you want to change the base?
Conversation
These were picked with the following command: > cat original-list-of-tests | sort -R | head -n10 | sort
This would cause us to end up with these results. That results in, for
[Edit: hah, beaten by @foolip by virtue of trying to write a table summarising the change!] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is fine as a placeholder change, but we should work out a way to get more coverage in a reasonable number of tests.
There's 45 other tests, so this takes us from 94.6% of the Web Compat focus area to 18.2%. Given the current scores of the whole directory, any subset of 10 tests here will alter the Firefox score for the directory by at least 5%. We could go up to 20 tests, which would be 30.8% of the focus area, and be able to better approximate the current scores. That said, I'm not trying to assert we change the current plan to move to 10 tests. Let's cross-reference this to web-platform-tests/rfcs#167 given subtests might help. It's also potentially related to broader questions about balancing scoring across features within a focus area for Interop, and this is not the first time this has come up. |
FWIW, per-element:
And per-property:
This doesn't make it super obvious what the one test for Safari should be, nor what the two/three tests for Firefox should be. |
These were picked with the following command: