The Oban Blog

Why is multivariate analysis and website optimisation only just here?


Well it’s not – multivariate analysis has been around for years and has been applied to other marketing channels for some time and with varying degrees of success; often with a great deal of success.

It’s not that multivariate analysis has not been a possibility, it just that it has not been adopted or at least widely adopted for website improvement.

I believe that there are four reasons for this:

  1. responsibility – who owns the responsibility for sales conversion? Not traffic to the site! Not usability of the site! Not the products and their respective price points! But, conversion of visits sales and sales profit figures?
  • is it the web-master?
  • is it the marketing manager or director?
  • is it your out sourced website development agency?
  • is it the sales department?
  • is it you?

Actually, it’s one of those difficult crossover areas and perhaps this is why it has taken so long for multivariate testing techniques to be adopted – and is still taking so long!

1.  trend – it hasn’t been trendy to adopt multivariate or A/B split testing techniques.
2.  tools – it is only recently that a number of tools have appeared on the market offering a point-and-click approach to multivariate website optimisation.
3.  cost – the tools that have been, and indeed are, available are costly (thousands £ per month). Google has introduced the free “Google Website Optimizer“, however it is really only suitable for certain situations and skill levels and has limitations.
4.  knowledge – there is a lack of knowledge in the website development and in the website marketing community in general. I believe that this will begin to change over the next few years.

In the main part I don’t think that those who have been tasked with looking after the marketing of a company and their products have the mathematical background that a multivariate approach would:

  • be in the “standard tool kit for marketeers”
  • be an easy sell for vendors that sell these technologies
  • be in the zeitgeist – but, as the buzz grows, so too will the vendors’ response and we will see more and more services and tools appear.

What should you do now?

If you are just starting to think about multivariate analysis and you are wondering what to do, then depending on who you are these are my suggestions:

I own a website and want to benefit from multivariate analysis:

At the moment there are quite a number of agencies offering services, particularly using the Google Optimizer tool. For a reasonably competent techie to build Google Optimizer into your site is not difficult. The cheapest option is a freelancer or a small website development company that focuses on functionality as opposed to graphical design see Wiredsussex for some examples.

My concern would be with two main areas of knowledge that you are buying from your consultant:
Do they understand how to set-up and manage the experiments. What is their statistical or experimental background that qualifies them to take your cash and play with your website.

How much experience do they have of content options or variants that might be presented to your website users? On what basis can they give advice on variations of content to best be tested?
My advice would be to focus on the knowledge and experience of the consultant as this is a “garbage in -> garbage out” situation. If your consultant is relying on blind luck – don’t use them.

I’m a marketeer and responsible for website marketing:

If sales conversion is not your responsibility now then once your boss catches on to how (very very) important it is then you are likely to be the one given the responsibility for improving it.

You are going to want to know what the ROI on adopting a tool or a consultant is and frankly it is almost impossible to calculate in advance and no-one is going to offer guarantees in this area.

I’d suggest approaching a consultancy or one of the tool vendors and ask for “best costs” on a trial period. Be prepared to lay some cash and time on the line to find out if multivariate can work for you, but draw a box around it:

“I’d like to put £5K to multivariate conversion optimisation and I’m going to measure the ROI”s

If you can see where your initial investment has gone, then you can evaluate a regular commitment.

I’m a webmaster and techie and I want to explore the possibilities of multivariate analysis:

There is lots to read out there and I am trying to compile as much as possible in this blog. If your company is considering multivariate optimisation then:

  • read up on Google Optimizer
  • try a split test for yourself (see below)
  • read up on the statistical methodologies – statistical significance is very important

Preparing a simple split test

Setup two landing pages (e.g. from Google Adword). Deciding how to vary them is the hard bit (try changing the look/feel and wording of calls to action). Give the pages near equal traffic. Set a cookie (content: pageA or pageB) so that you know which page the user has been presented with. Using the cookie make sure the user gets the same page when they return to the site. When a purchase is make write the timestamp, cookie and purchase value to a log or DB table. Import the log or table into a spreadsheet on a regular basis so you can see which page promotes more sales and/or higher value sales.

If, based on this, you think that this approach might work for you, don’t adopt a hacked approach (ie extending the above) – there are some good tools available and the embedded knowledge of the development companies is a big part of what you are buying.

I’m a website development agency or SEO company and want to incorporate multivariate optimisation in to my offering:

My advice would be to take the partnering approach. Partner with consultants or with one of the product companies. Only a small percentage of sites need this in a way that you can make a significant profit – partnering is a way the completes your services without a great overhead.