The Oban Blog

The Nth degree: what is ABn Testing?

To AB or not to AB? That is the question.

Conversion optimisers for some time have fallen into two main camps – those who prefer AB testing and those who prefer multivariate (MVT). For some, AB testing offers an opportunity for a quick, clean result via a shoot out between the original design and the challenger. Following a methodical path, it analyses a single change to the website, gathering feedback on that specific change in isolation.

Meanwhile, advocates of MVT prefer the possibilities of testing various changes at once. Finding winners and losers, as well as, identifying interactions between on page elements. MVT usually takes longer, needs more test traffic and sometimes provokes more questions than it answers. Nevertheless, it offers richer data that sometimes reveal a change in language or verbiage that only works in combination with a change of position or colour.

So what is ABn? The third option is more than two competing test methodologies. ABn testing occurs when you want to test more than an individual change at one time (in stats, n denotes a variable between one and any number). In a test brainstorming session contradictory views might emerge between optimisers over why they feel a page is underperforming, leading to at least three different designs. In this case conducting an ABn test will certainly help provide answers to opposing opinions.

AB test example

In the AB test sample below the online gaming platform, Poker Heaven, chose a straight shootout between the original and a new design, to identify which design increased more registrations. In comparison, version A is text heavy and features access to a wide range of games for new and returning visitors. Version B displays a simpler cleaner design, with a prominent gold call-to-action button and a thousand euro incentive. Consequently, version B won the registration test effortlessly. If, however, we were to apply an ABn test to this, it would enable Poker Heaven to test a third design. This could then combine the drive for new registrations with fresh content to measure returning customers.

ABn test example

The following ABn test example shows how the leading online travel organisation, STA Travel, tested a few designs to increase searches for flights. The original (version A) displayed a homepage dominated by a prominent flight search box on the left hand side. Version B dispensed the flight search from the homepage completely, whilst versions C, D and E, experimented with a slimmer vertical flight box aligned left and right, and with a horizontal version of the box below the fold.
This test measured the users’ propensity to use flight search as well as site interaction. Interestingly, removing search completely encouraged a huge uplift in visits to product pages that previously hadn’t seen much traffic- such as ‘Tours and Deals’ and ‘Discounts’. In this case, the winner was version D which encouraged a better flow around key areas of the site and generated more revenue.

In conclusion,  ABn offers another dimension: the ability to settle arguments between competing opinions, action a broader set of test goals, isolate a specific on page element in various iterations, maximise test time and check a web redesigns’ performance. So, the next time you’re looking at your test programme remember two may be company, but sometimes three (or more) might be the crowd you need!