Incorrect speed index calculation

[AAZANOVICH]

According to SpeedIndex documentation, speed index is calculated as a sum of interval scores
IntervalScore = Interval * (1.0 - (Completeness/100)). So, on the screenshot below same website tested twice (different revisions, 1st which supposed to be faster, and 2nd supposed to be slower). For the first test we can see that frame snapshots contains visual content almost immediately and for second test content appears only at the end.. However SpeedIndex for second test is lower than for the first one.. which doesn't make any sense..

[patrickhulce]

Do you have a publicly accessible URL with repro steps @AAZANOVICH?

The frame snapshots do not share a global timeline and the FCP value in the first case is much much higher than the second case (indicating that the first frame with content in screenshot A is actually at 6s while the second case the last screenshot is at ~2s) which perfectly explains why Speed Index is higher.

This appears to just be a case of performance variance.

[paulirish]

Adding on to what @patrickhulce said... the lack of a timescale on the screenshots indeed make this tricky to interpret. Sorry about that.

[AAZANOVICH]

I can tell for sure first page loads way faster visually.. I cannot share source code, but I will record a video with timing later

[AAZANOVICH]

Ok, First test screenshot_1 SpeedIndex 6+s, however however as you can see on a timeline it appears quickly. 2nd test SpeedIndex 3.1 s, but on timeline content start to appear after 1s+.

Any ideas or anything else I can provide to help you, I cant share code, but I can even do a video call and share my screen. Or we can even meet somewere I from bay area ).. I work on performance and this is really annoying.

[patrickhulce]

Thanks for the trace screenshots! As for extra information, the bug template environment questions are a good start i.e. how are you running lighthouse, what channel, OS, what settings are you using, etc

My best guess for what's going o based on your trace times is that this looks like simulated throttling and what appears to be a bug in FCP. If this is indeed simulated throttling, does the same issue occur with "applied" throttling (CLI --throttling-method=devtools)?

[AAZANOVICH]

Running test from Chrome Dev Tools -> Audits tab
Chrome: Version 78.0.3904.97 (Official Build) (64-bit)
OS: MacOs 10.4.06
Network throttling: 150 ms TCP RTT, 1,638.4 Kbps throughput (Simulated)
CPU throttling: 4x slowdown (Simulated)
Device: Mobile/Desktop
Running on localhost
Node version: 10.10
This is an Angular 6 app.
Applied throttling doesn't help, same issue, a bit different numbers.

Not sure what do you mean by bug in FCP. If you look at second screenshot and see that FCP appears at about 200ms but screenshots are empty, this is not an issue.. There is a spinner on the page which affect FCP. However there is something strange with FCP for sure, I even removed spinner from index.html. See screenshot, on a timeline FCP appears at 495.4 ms, however FCP on report is 6.5s..

[AAZANOVICH]

BTW, you can put some log statements into command line version and I can run and provide you logs you need..

[Lofesa]

@AAZANOVICH @patrickhulce
Sorry for spamming this....
Maybe is the discordance between audit panel and performance panel. When in audit panel if you click in to the "View Trace" button you go to the performance panel but here the numbers are for unthrottled fecht.
In performance panel you can set 3G Fast and 4x slowdown and the numbers are not the same but like the audit panel.

[patrickhulce]

There is a spinner on the page which affect FCP

Ah, I could not see the spinner from the screenshots that certainly explains a little more thanks! @AAZANOVICH if you could run the CLI with --save-artifacts and provide the .trace.json and .devtoolslog.json files for the runs we'd be able to dig into the differences here and see what's going on. Something definitely seems off if it's happening in applied throttling modes too, so we'd love to get to the bottom of it thanks for your help! :)

[AAZANOVICH]

@Lofesa why numbers (when clicking view trace) are for unthrottled fecht.? If it is the case it is very confusing. As well as when dropdown menu says 3g fast, but same speed in Audits is 4g slow ..

@patrickhulce seems like "--save-artifacts" is outdated it doesn't generate artifacts. I ran it with --save-assets this looks like generated files you wanted. See attached archive.

Command i used: lighthouse https://localhost.prudential.com:9000/ --verbose --chrome-flags="--headless --ignore-certificate-errors" --save-assets --throttling.cpuSlowdownMultiplier 4 --only-categories=performance
test1_assets_2020-02-05_12-51-45.zip

[AAZANOVICH]

Here is a second run for not_optimized version, which visually runs slower (you can see it on snapshots as well), but having better SpeedIndex
test2_not_optimized.zip

Also what is exact param match for commandline tool which is equal to Chrome Audits tool: Mobile, Slow 4G Simulated, 4x CPU slowdown, Clear Session Storage . ????

[Lofesa]

@AAZANOVICH See #9971

[AAZANOVICH]

@Lofesa I see, thanks.

[patrickhulce]

Thanks very much for those files @AAZANOVICH! Apologies for the incorrect flag --save-assets was indeed what I meant, kudos for figuring it out 👍

Now to Speed Index computation...

This mostly comes down to a flaw in the Speed Index definition that you only get credit for exact color values being present, and because the loading overlay on this page is white with alpha, the very early presentation of this screen

is basically given 0% completion credit.

There was an alternative metric called "Perceptual Speed Index" that attempted to fix this issue with "Speed Index" but it had its own serious flaws and never really saw industry-wide adoption, so Lighthouse standardized on "Speed Index".

What this means for this page unfortunately is that this is somewhat of a can't fix :/ If one were to make changes to please Speed Index, then the loading spinner would just appear on top of the content without the white alpha, but if the content really isn't interactable during this timeframe then IMO that's a step backwards for UX and you should stick with the current approach.

Sorry this isn't a more satisfying outcome for you @AAZANOVICH, but the good news is that in 6.0 the new balance of metrics moves a bit away from leaning on speed index and the weighting of new metrics LCP and TBT would strongly favor the optimized version and be reflected better in the performance score.

[AAZANOVICH]

@patrickhulce ok, based on what you said, if I remove loading overlay I should get better SpeedIndex. Let me check that, just to confirm it is the case and not something else. Because if you compare both runs, you still can see that full page was rendered (without loading overlay) still a bit faster, compared to non optimized, but SpeedIndex is 6.6 compared to non optimized 6.0 (which can variation, but lets see..)

[patrickhulce]

Yes please confirm using any observed speed index methods (--throttling-method=devtools or --throttling-method=provided). While the change should also have some effect on the default simulated throttling, the speed index there is approximated and might still have some issues.

[AAZANOVICH]

Looks like you right, overlay obviously making an impact. I think we can close this one.. Not sure that it was the only issue, but one of them for sure, I need to do more testing..

How do you read "timestamp": 49663.419855 in your log ? What is this 49663.419855 ?
Is there any documentation which can help analyze these logs, so I can troubleshoot myself better ?

"method": "Page.lifecycleEvent",
"params": {
"frameId": "34A28F44BA54CF4761B459560900A52D",
"loaderId": "88DAC423358A3F7882EC2048C3F94E2A",
"name": "networkAlmostIdle",
"timestamp": 49663.419855
}

[patrickhulce]

Glad to hear you're seeing the effects!

The timestamp in the devtoolslog is a main-thread timestamp that can only usefully be judged in relation to the first timestamp. It's pretty difficult to manually inspect the logs for information and most debugging involves parsing the logs in some way before they're useful.

Protocol & Trace Notes in Architecture Documentation
Trace Processing Code
Devtools Log Network Processing Code

I'm going to go ahead and close this one, but feel free to comment again if you think there's something to be addressed or have anything to share.