Why your PageSpeed score changes between tests

The short answer

A PageSpeed Insights score is the result of one simulated page load. Lighthouse, the tool inside PageSpeed Insights, says so directly: scores change because of inherent variability in web and network technologies, even if nothing about the page was changed. It tells you to run it several times before you draw conclusions.

That does not make the score useless. It means a score is a measurement with some noise in it, like a thermometer that reads a little differently each time. The skill is knowing how much of a change is noise.

Seven sources of variance

The Lighthouse documentation lists these sources and rates how much each one matters. It also rates how likely each is for a person running a test on their own computer, in PageSpeed Insights and in a controlled lab. The last column below is the rating for PageSpeed Insights.

Some of these are in your hands. Page nondeterminism is a property of your site, and the web server is yours too. Local network variability and client hardware matter when you run Lighthouse on your own computer. The documentation rates them unlikely in PageSpeed Insights, which is a hosted test that runs on Google's side.

What to look at first when the score moved

Work through these in order. They are the most common causes of a change you did not make.

1. The page was different. Ads, banners, consent pop-ups, A/B tests, live chat widgets and "recommended for you" blocks can change from one load to the next. The Chrome documentation puts A/B tests and changes in the ads being served first on its list of causes. If the page is not the same page, the scores will not match.
2. The server was slower. A busy server, an empty cache or a backup running in the background can all raise the wait for the first byte. See how to reduce server response time.
3. A third-party script changed. Chat, analytics, tag managers and ad tags are loaded from other companies' servers and can change without notice. See third-party scripts.
4. You tested a different way. Mobile and desktop are separate tests with different settings. A different tool, a different Lighthouse version, a logged-in browser or a cold cache all give different answers.
5. The test machine was busy. Browser extensions that inject JavaScript or modify requests, and antivirus software, appear on the Chrome documentation's list. The Lighthouse variability document adds running more than one test at the same time on one machine.
6. The tool changed. The weights of the five metrics have changed over time as the Lighthouse team has updated them, so a new Lighthouse version can move a score without any change on your site. The footer of a Lighthouse report names the version that produced it.
7. Routing on the internet changed. This is also on the Chrome list. It affects timing and is outside anyone's control.

Why two tools never agree

A score is not a property of your page. It belongs to a combination of the page, the test machine, its settings and the tool version. Change any of them and the score can change.

By default Lighthouse uses simulated throttling. It loads the page once without throttling, records what happened, and then calculates what a slow phone on a slow connection would have experienced. That makes it fast and repeatable, and it means that the result is an estimate. Its mobile settings are a network with a 150 millisecond round trip and 1.6 megabits per second of download, and a processor slowed down four times. Lighthouse describes these as roughly the slowest 25% of 4G connections and the fastest 25% of 3G connections. Your own phone on your own connection may be much faster, which is why a site can feel fine to you and still score low.

The same page can also look different in the two halves of PageSpeed Insights. The top half is field data from real Chrome visitors over 28 days. The bottom half is a lab test of one load. They answer different questions, as lab data and field data explains.

How to test so the number means something

Lighthouse's documentation gives specific advice:

• Run it more than once and use the median. The median of five runs is twice as stable as one run, according to the Lighthouse team. Use the middle value, not the best one.
• Do not run tests at the same time on one machine. Several runs at once compete for the processor and distort each other.
• Use a machine with enough power. Two dedicated cores at least and four recommended, with 2 GB of memory at least. Avoid shared or "burstable" servers for automated tests.
• Keep the page the same. Test the same version of the page. Pause A/B tests while you measure.
• Remove outside influences. Use a private window with no extensions, and a device that is not busy with something else.
• If your computer is a poor test machine, use a hosted tool. PageSpeed Insights and WebPageTest run the test for you.

From the command line, Lighthouse CI can run the test five times:

npx -p @lhci/cli lhci collect --url https://example.com -n 5

It can also pick the median run for you, as the Lighthouse documentation shows. When you compare two versions of a page, test both the same way, at about the same time of day, with the same number of runs.

Which changes are real

A change is probably real when it shows up in all or most runs, when a specific metric moved by a clear amount instead of the score moving by a point or two, when it matches something you changed on a date you know, and when the real-visitor data later moves in the same direction.

A change is probably noise when it appears in one run and disappears in the next, when the metrics that make up the score hardly moved, or when it happened while a cache was empty, a deploy was in progress or a campaign changed the page.

Do not set a goal of one exact number. The Chrome documentation suggests thinking of performance as a distribution of scores instead of a single number. Think of it as a range, and aim to move the range.

A short routine you can repeat

1. Pick the page and the device setting, and write both down.
2. Run five tests and record the median score and the median of each of the five metrics.
3. Change one thing.
4. Run five tests again and compare the medians.
5. After a few weeks, check the field data for the page. That, not the lab score, shows whether visitors feel the difference.

Check your own page

Run the free audit more than once and compare the numbers of the five metrics, not only the score. If the score moves but First Contentful Paint, Largest Contentful Paint and Total Blocking Time stay about the same, you are looking at noise. If one of them moves a lot between runs, find out what in the page or the server is unstable.

Sources

• Lighthouse score variability: the seven sources, the hardware advice and running Lighthouse several times.
• Lighthouse performance scoring: why the score fluctuates and how the weights have changed over time.
• Network throttling in Lighthouse: the mobile settings and the default simulated throttling.