Automated speed tests and performance budgets

Why automate

Without a check, speed work decays. A fix is made, the scores go up, and over the next months new content, plugins and scripts pull them down again. Nobody notices until a customer or a search report does.

An automated test makes speed a property of every change. It tells you which change cost what, while the change is fresh and small enough to fix.

What a performance budget is

MDN defines a performance budget as a limit to prevent regressions. It can apply to a file, a file type, all the files on a page, a specific metric, a custom metric, or a threshold over a period of time. MDN groups the goals into three kinds:

• Timing-based, such as Largest Contentful Paint or Time to Interactive.
• Quantity-based, such as the amount of JavaScript or the total size of images.
• Rule-based, such as a Lighthouse score.

The primary goal is to prevent regressions, though budgets can also show trends and expose needs, such as a large library that has smaller alternatives.

Two levels

MDN says a budget should have two levels:

• Warning. Lets you act early and plan to clean up, without blocking work or deploys.
• Error. An upper limit, where a change will have a negative and noticeable effect.

Where the numbers come from

Start from your visitors. MDN's advice is to measure the devices and connection speeds your users have, with several tools, and to turn those time-based goals into file-size budgets. It also says budgets should be dynamic: you may accept a feature that costs load time if it is meant to raise engagement.

For a first set, the Core Web Vitals thresholds are a natural error level: 2.5 seconds for Largest Contentful Paint, 200 milliseconds for Interaction to Next Paint, 0.1 for Cumulative Layout Shift. Put a tighter figure on the warning level. For sizes, look at the current page and set the budget at or just below it, so that it only moves in the right direction. See page weight and requests.

File-size checks in a build, such as the performance warnings in a bundler, are a first line of defense. MDN adds a caution: size is hard to translate into time, because a development build may lack the third-party scripts and the CDN optimizations of the live site.

Why one run is not enough

Lighthouse's own variability documentation starts with this: scores change because of variability in web and network technology even if the code has not changed, so run it multiple times and be wary of variability before concluding that a change made a difference.

It lists the sources. The biggest are page nondeterminism (for example, an A/B test or different ads), local network variability, client hardware variability and resource contention from other programs on the machine. Its simulated throttling reduces some of these, and nothing removes page nondeterminism. If the page differs between runs, the numbers will too, so test the exact same version of the page.

Its advice for stable results in an automated setting:

• Use adequate hardware: at least 2 dedicated cores, with 4 recommended, and at least 2 GB of memory.
• Avoid function-as-a-service platforms and burstable or shared-core machine types.
• Do not collect several Lighthouse reports at the same time on the same machine. Running in parallel distorts the results, and the document recommends scaling horizontally with more small machines instead.
• Run several times and take a representative value.

Lighthouse CI runs three times per URL by default, and you can raise it with the numberOfRuns setting.

Lighthouse CI

Lighthouse CI is the Lighthouse team's tool for running Lighthouse in a build. According to its documentation, after setup your build system runs Lighthouse on your project's URLs on every commit, asserts that chosen audits pass, and uploads the reports for inspection.

The steps

1. Prerequisites. Your code is in git, your changes go through a continuous integration system such as GitHub Actions, the build can produce production assets, and you have either a command that serves them or a static site.
2. Collect. A file named lighthouserc.js at the root of the repository holds the settings. The command lhci autorun runs the sequence. It finds the static files, or starts your server, and runs Lighthouse on each page.
3. Upload. Reports can go to "temporary public storage" or to your own Lighthouse CI server. The documentation warns that the temporary storage is public: anyone with the URL can see the reports. Read its terms first, or use your own server.
4. Assert. Add rules, and the build fails or warns when a rule is broken.

The documentation recommends starting slowly. Collect results first. Add assertions later, once the scores are a little higher.

Assertions

An assertion names a Lighthouse audit and a level in the style of a code linter: off, warn or error. You can check the score, the number of items or the numeric value, with minScore, maxLength and maxNumericValue. A preset such as lighthouse:recommended starts you with the Lighthouse team's list of audits.

A practical approach from the documentation: start with the preset, look at the failures in the build log, switch those audits off, and treat that list as a burndown list of issues to fix.

For several runs, an aggregationMethod decides which value counts: the most favorable, the least favorable, or the run judged most representative. For different pages with different limits, assertMatrix applies different assertions to different URL patterns.

An example

This configuration is an example for this guide, not a recommendation of particular numbers:

module.exports = {
  ci: {
    collect: {
      url: ['https://staging.example.com/', 'https://staging.example.com/pricing/'],
      numberOfRuns: 5,
    },
    assert: {
      assertions: {
        'categories:performance': ['warn', {minScore: 0.9}],
        'largest-contentful-paint': ['error', {maxNumericValue: 2500}],
        'cumulative-layout-shift': ['error', {maxNumericValue: 0.1}],
        'total-blocking-time': ['warn', {maxNumericValue: 200}],
      },
    },
    upload: {target: 'temporary-public-storage'},
  },
};

Use a staging or preview address, and the same settings every time.

The limits of a build-time test

• It tests a build, not your visitors. A staging site may lack the live site's tags, CDN and caching. The result is a regression detector, not a prediction of the real numbers. Check real-visitor data too. See real user monitoring.
• It tests a few pages. Choose the ones that matter: the home page, the top landing pages, a product or article page, and the pages with the most scripts.
• It tests a signed-out visitor. Carts and logins need special handling. See the WooCommerce checkout guide for how to test with a cart.
• Scores are noisy. Alert on a sustained change, not on a one-point drop.

If you have no build process

A site on a hosted builder or a WordPress site updated through a dashboard has no build step to hook into. The same discipline works by hand or on a schedule:

1. Pick five pages: home, two landing pages, one content page, one form or cart page.
2. Test each with the same tool and settings once a month, and after every plugin, theme or script change. Take the median of several runs. See why scores change.
3. Write the numbers and the date in a sheet, with a note of what changed.
4. Set a rule, for example that Largest Contentful Paint over 2.5 seconds on any of the five pages means someone looks into it this week.
5. Check the real-visitor data in PageSpeed Insights each month. See the Chrome UX Report.

Online monitoring services that run tests on a schedule and send alerts also exist. Compare what they measure and how often before you rely on one.

When a check fails

1. Run it again, and check that the page and the settings are the same.
2. Compare with the last good run: which change went in between?
3. Open the report and find the audit that changed.
4. Fix it, or decide on purpose to accept it and change the budget, with a note of why.

Mistakes to avoid

• Budgets set from wishes. Base them on what your pages and visitors are today.
• Errors only. Warnings keep teams working. Errors should be rare and real.
• A single run. Use several.
• Public reports by accident. Know where the reports go.
• Testing in different conditions each time. Fix the machine, the settings and the version of the page.
• Alerts nobody reads. Decide who looks, and when.

Check your own page

Run the free audit on your five key pages and record the results. That is your baseline. Set a warning level a little above each result and an error level at the Core Web Vitals threshold, and repeat the test on a schedule.

Sources

• Lighthouse CI getting started and configuration: the setup, the number of runs, the storage warning, presets, assertion levels and options, and assertMatrix.
• Score variability in the Lighthouse documentation: the sources of variability and the advice on hardware and repeated runs.
• Performance budgets on MDN: what a budget is, the two levels and how to set one.