Big mistake with Google Website Optimizer!

From the desk of Roy Furr, Thursday, January 24th, 2008

Here’s an email I received recently from someone who watched these Google Website Optimizer videos:

Just watched the 2nd video posted on Multivariate Testing for Google Optimizer.

Thanks again for the information. Again, the video was integral in allowing me to set up the experiment in a little time as possible.

In fact, I successfully installed all code all tracking code the first time. I now have a home page with 83 variations! I guess I went nuts.

Very efficient!

I felt bad for the guy because he’d put in a lot of work. But because his website is not super high-traffic, I had to burst his bubble. (It’ll save him some pain down the road.)

Here’s my reply:

Glad the video was helpful! But…

83 variations is a lot if you don’t have tons of traffic. You’re going to need between 100 and 1000 people per variation in order for you to get good results. That means as many as 83,000 visitors (maybe even more than that!) will have to see your website before you know the results of your test.

Unless you do get that type of traffic (or plan to spend enough to start getting it) you’ll probably want to look at what’s really important to test, and re-start your test focusing on the top 2 or 3 things you want to test, with no more than 3 variations each.

What I often do when I use their multivariate tool is test 3 headlines and 2 each of 2 other variables. That gives you 12 total variations and gets you results quick enough you can move on. Much more than that and you want to start getting into the advanced stuff like Design Of Experiments (DOE) and Taguchi to really maximize the improvements you can get from the same amount of traffic — but that goes beyond what Google Website Optimizer does natively (Bob Bly will be releasing an ebook I wrote soon that shows you how to do it with Google Website Optimizer).

Sorry if this throws a wrench in your cogs… I just don’t want you to be bummed a few months from now when you haven’t gotten conclusive results on your tests yet!

Let me know if you have questions, or if there’s anything else I can do for you.

This is a common mistake. With Google Website Optimizer’s multivariate testing, you can create and test over 4,000 combinations of elements on a web page. But… since each combination will need between 100 and 1,000 visitors (or more) before you can really learn anything from it, you’re going to need a lot of traffic to make this successful — as many as 4,000,000 visitors!

You’d be better off testing 6 headlines against each other (and probably get better results quicker). This way you have 6 total web pages to test, and in 6,000 visitors or less you’ll know which is the most effective! Then you can test other elements in the page.

Another alternative that gives you 6 options is 3 headlines and 2 of another element. That’s a good test as well. There are a lot of options that keep your total number of tests around 15 or below, which is a good number for most websites. This gets results quickly, with a smaller amount of traffic.

Much more than that and you may want to call me for Taguchi testing. (I can help with the smaller split tests and multivariate tests too, if you’d like.) 541-543-1438

Let me know what I can do for you!

- R

P.S. - Click here for more Google Website Optimizer Information and How-To Instructions

Tags: Google Website Optimizer videos, Multivariate Testing, Google Optimizer, experiment, home page, website, visitors, traffic, multivariate tool, Design Of Experiments, DOE, Taguchi, Google Website Optimizer, Bob Bly, ebook, headlines, Taguchi testing, split tests, multivariate tests

This entry is filed under Google Website Optimizer. You can follow any responses to this entry through the RSS 2.0 feed.You can leave a response, or trackback from your own site.