Speed Lab: three ways to slow a test down

The question

The same page can score differently in two tools, and the Speed Lab's own results depend on how the lab slows the network down. Lighthouse's documentation lists four styles of network throttling. Simulated throttling loads the page once at full speed and calculates what a slow connection would have done. Request-level throttling, which Lighthouse calls devtools, makes the browser delay and limit each request. Proxy-level tools sit between the browser and the server. Packet-level tools slow the network itself, and WebPageTest uses one. The documentation says that on a real mobile connection latency acts at the packet level, not the request level, and that request-level throttling is a decent approximation and not a substitute for packet-level tools. This test puts numbers on the difference, using pages from the Speed Lab.

The three ways I used

The 562.5 ms of applied throttling is 150 ms multiplied by 3.75, which is how Lighthouse converts its network settings for request-level throttling. The shaped network uses the figures of Lighthouse's own mobile preset: a round trip of 150 ms and 1.6 Mbit/s down. The proxy is a proxy-level tool, which the documentation calls decent but not ideal because it does not touch UDP. A packet-level shaper such as @sitespeed.io/throttle needs administrator rights and slows the whole machine, which is why I did not use one. The code for mine is in the test of many small files.

The same changes, three answers

Each row is one change, measured the same way under each method: the median of the page that has the cost minus the median of the page that does not, in milliseconds. A minus sign means the change made the page faster. The applied figures come from the earlier tests, with nine runs per page. The shaped and simulated figures come from five runs per page.

The last row compares Node's HTTP/2 server under applied throttling with Hypercorn's under shaping. The first four rows come from the lazy-loading and discovery tests, the next three from the script and critical CSS tests, the next one from the font test, and the last four from the test of many small files and the priority test. The font row shows that some results do not move with the method: a preload made the first paint later by 108 ms and 103 ms, and I cannot explain it under either.

Three patterns show up.

A chain of requests: the usual lab method exaggerates

The delay of 562.5 ms is added to every request. When one request has to wait for another, as an image waits for the style sheet that names it, the lab adds that delay once more for each step, and a connection with a round trip of 150 ms adds less. The costs of a late main image, of lazy loading and of an extra step before the first paint were all smaller on the shaped network. Lazy loading cost 311 ms under applied throttling and nothing on the shaped network, a CSS background 298 ms against 16 ms, a script-inserted image 587 ms against 258 ms, and inlined critical CSS saved 786 ms against 418 ms.

For the lazy image and the CSS background the cost fell into the noise. These pages are bound by bandwidth: a main image of 189 KB takes about a second to cross a link of 1.6 Mbit/s, and it arrives last however it is found. Starting late costs time only if the link is left idle in the meantime, or if other files get in front. The lab's 562.5 ms leaves the link idle for longer, so it finds a cost that a visitor on a shorter round trip would not.

Bytes and slow start: the usual lab method can understate

A script in the head that blocks the page cost 486 ms under applied throttling and 637 ms on the shaped network. The script is 59 KB and was sent uncompressed. On the shaped network a new connection starts slowly, so a file of that size needs several round trips to arrive, and the page waits for all of them. Applied throttling does not model slow start. The same reason made the bundled style sheet slow in the test of many small files when the files were not compressed.

Connections and the order of files: only the shaped network shows them

Merging 22 files into two over HTTP/1.1 made the first paint 309 ms sooner under applied throttling and 303 ms sooner in the simulation, and 161 ms later on the shaped network, where twelve small style sheets on six connections started faster than one large one. The test of many small files explains it. And when I marked six thumbnails as low priority over HTTP/2, the main image finished 264 ms sooner on the shaped network, from a server that follows the browser's priorities. Applied throttling showed no difference at all, because it limits the download after the bytes have arrived, and the simulation has no model for it.

The simulation: a calculation with edge cases

On these small local pages the simulation gave several answers that do not match the other two methods or the page's own timeline. Lazy-loaded images looked faster than normal ones, by 902 and 1,051 ms. A script at the end of the body delayed the first paint by as much as one in the head (1,425 ms against 1,386 ms), although the page painted before the script in both other methods. The two ways of loading the rest of the critical CSS differed by 248 ms, and the preloaded one gained nothing, where the other methods found them equal.

The documentation warns that the simulation is roughly as accurate as request-level throttling for most sites but has edge cases, and that a deep investigation should use packet-level tools. I did not look for the causes inside the simulator. It agreed with the shaped network on the blocking script (634 ms against 637 ms), and on the many-files test it matched applied throttling for HTTP/1.1.

What each way is good for

• Simulated is fast and steady. It is the right choice for comparing one page before and after a change, and it is what PageSpeed Insights uses. It is a calculation from one fast load, so it can miss what a server does differently when a connection is slow.
• Applied shows what Chrome does when requests arrive slowly: how it queues them, which priority it gives them and how many connections it opens. It is steady and it needs no extra tool. It overstates the cost of long chains of requests, it does not charge for opening connections, and it cannot show the order in which a server sends files.
• Shaped (or any packet-level shaper) is the one to use when the question is about connections: HTTP/1.1 against HTTP/2, the order of files, the first connection to a new host. It is slower to run and has settings of its own, such as the size of the starting window.

When a result matters to a decision, run it with two methods. If they agree on the direction, trust the direction. If they disagree, find out why before you act.

A trap: the first paint that arrived late

When Lighthouse applies no network throttling, as it must when a shaper does the work, headless Chrome 153 often reported the first paint late. In my first 18 runs through the shaper, 14 reported it at about 2.5 seconds whatever the network did, at times that had nothing to do with the files. A page with no shaping at all did it once in six runs. Switching off Chrome's frame rate limit with --disable-frame-rate-limit removed it in every run I kept. --disable-gpu-vsync alone did not. If a first paint looks oddly constant from run to run, check for this.

What this test does not show

• A small set of tests. Five runs of each page in the shaped and simulated columns, on one machine and one browser build.
• A packet-level shaper. I did not compare my proxy with one, so it may differ from what such a tool would show.
• Tests of bytes. I did not run the compression and image format tests again on the shaped network. Their effect comes from the number of bytes, and all three methods use about the same download rate, so I expect similar answers. I did not check.
• Real visitors. All of it is lab data. See lab data and field data.

What to do

1. Know which method your tool uses. PageSpeed Insights and the Lighthouse command line simulate by default. The Lighthouse panel in Chrome DevTools offers devtools throttling in its settings.
2. Compare like with like. Two reports from different tools, or from the same tool with different throttling, are not comparable.
3. Be careful with effects that depend on a chain of requests, such as a preload, a lazy image or an inlined style sheet. The lab figure will often be larger than a visitor's. The direction usually survives.
4. For questions about connections, use a shaper. Lighthouse's documentation shows how to switch off its own network throttling and keep the processor slowdown.
5. Read field data when you can. It has all of the real network in it. See lab data and field data.

Mistakes to avoid

• Quoting a lab number as if it were a visitor's.
• Comparing a PageSpeed Insights score with one from a different tool without asking how each slows the network.
• Testing a connection question with request-level throttling.
• Trusting a first paint that never varies.
• Judging a change by one method and one run.

Check your own page

Run the free audit on your page and note the first paint. Run another tool on the same page and compare, then read how each one slows the network down. If a fix matters, test it with two methods before you build more on it.

Sources

• Network throttling in the Lighthouse documentation: the four styles of throttling, the mobile preset, and how to run Lighthouse with a shaper in front of it.
• Variability in the Lighthouse documentation: why repeated runs differ and how to reduce it.
• The earlier Speed Lab tests, which carry the figures in the table.