7 November 2019

Testing is important. Regularly assessing which version of a web page or website will deliver better engagement or conversion rate enables you to improve continually in a virtuous cycle of test, optimise and retest.

But what if the testing itself is affecting the results?

If you implement A/B testing on the client side, using JavaScript, you run the risk of slowing the page down.

This is partly because you don't have precise control over when that script will be loaded or how long it will take to execute. It will have to compete with other resources on its way to the end user's device. And when it gets there, how capable is that device? Is it a brand new iPhone or a five-year-old mid-market tablet?

Testing solutions will also often hide the page until the experiment is ready (typically by setting its opacity to 0). There is a good reason for this – you don't want a default page to appear, only to be replaced moments later by an alternative version. But it still introduces a delay – the page is slower than it needs to be. Another side effect of this is that when the page is ready, it tends to be rendered all at once rather than building up gradually, one or more element at a time. There's nothing wrong with this per se. But it's another example of the testing process affecting what you're testing. The upshot is that when you roll out one version or another, it's likely to load differently. In other words, your roll-out version will be different from your test version.

A/B testing – an example

I recently came across a site that used Google Optimize for testing. The page was only revealed after two scripts had loaded and executed, with the first script fetching the second.

Here's how is looks in an extract from a waterfall chart (tested with Webpagetest using Chrome at 5Mbps):

I've cropped the chart for readability, so what you can't see is that the first script is the 34th asset to start loading. In fact, it doesn't begin loading until 1.8s into the test.

Why is this?

Well, one reason is that it's loaded asynchronously. In other words, parsing and rendering of the page are able to continue while the script is loading. While this is often helpful from a performance point of view, browsers will also tend to give asynchronously loaded scripts lower priority (and in this case, Webpagetest reported a priority of LOWEST). So here, despite the fact that the script was added at the top of the page (injected by an inline snippet), it still loaded late. Which meant the second script also loaded late. Between them, they took around 1.5s to load and execute, eventually allowing the page to display at 3.4s.

Possible solutions

Preconnect

In this test, around half a second was spent making the initial connection to the domain for the first script. Preconnecting to the domain using <link rel="preconnect" ...> should allow the connection to be established well in advance, so by the time the browser gets around to loading the script, it will already have a connection open. In this case, the page could have displayed at 2.9s rather than 3.4s – that's a pretty big win.

Preconnect is a resource hint, not a directive, so there are no guarantees about what the browser will do with it, but there's not a whole lot to lose either.

Limiting the time the page is hidden

What if your experiment fails to load for some reason? Or if there is a very long delay. By default, Google Optimize applies a 4-second timeout – it won't hide the page for longer than 4 seconds. But that's still quite a long wait. Fortunately, this setting is customisable, and you can reduce the time if you wish. The disadvantage is that if you set a very short timeout, you may not get meaningful results from tests under poor network conditions, which could skew your overall results.

Loading the first script synchronously

Asynchronously loaded scripts are useful because you don't normally want scripts to hold up rendering of the page. But as we've seen, here the script delays rendering whether it's loaded synchronously or asynchronously. At the very least, it might be worth testing to see whether loading the script synchronously increases its priority and allows it to load earlier.

Switching to server-side experiments

There is always an element of unpredictability when you hand control of your A/B testing over to the client. So why not run your tests server side? The good news is that this is offered by several providers, including Google Optimize.

All in all, while setting up A/B testing is relatively easy, doing it in such a way that minimises the performance impact and delivers meaningful results can be more challenging. It may require some fine-tuning and going beyond simply sticking to accepted best practices in order to deliver better tests and better experiences.