Are you basing copy and design changes on your gut feeling or what your boss prefers? Stop that! There is a better way to optimize your website.

By using a tool for A/B and multivariate testing, you can see what your visitors prefer and let the numbers speak. This allows you to find out which website tweaks make your visitors take action and increase sales.

Don’t have a tool for testing? No sweat! There are numerous tools to choose from, but I recommend that you first try Google Website Optimizer. It is free (get it here), and with it, you can find out what features you really need and which are just nice to have.

When you get started with testing, you need to keep some things in mind.

1. Remember that best practice is somebody else’s practice
You can look at what others have been doing successfully, but don’t assume what worked for them will work for you. Depending on your audience and context, results will vary. An example: some experts say that one-step forms are always more efficient, whereas others claim that multi-step forms are superior. So who’s right? Well, they’re equally right and equally wrong. Tests have found that sometimes one-step forms work better, and sometimes multi-step forms do. Don’t assume. Test what works best for your audience and context. Oh, and if someone says that red buttons are better than purple ones, don’t believe them. There is no universally superior button color, but you may want to test contrasting colors.

2. Be bold and break rules
You may have heard about Google’s testing of 41 shades of blue for the toolbar on Google pages, but unless you work for Google, forget about it. Minor changes will typically lead to minor improvements, and to find out if those minor improvements are statistically valid, you’ll need massive amounts of traffic. If you want results, you’ll have to make more drastic changes than that. You will often have to bend, if not break, company design guidelines in order to achieve substantial improvements. While doing so, it’s important to work closely with brand managers, so you do not step on anyone’s toes. Examples of what to test include images, tone, and functionality.

3. Don’t be greedy (limit your test)
Unfortunately, not everyone has the benefit of getting a lot of visitors, and traffic volumes may limit you to running an A/B split test. Maybe you think changing the primary headline, the main copy text, an image, the call to action, and a button will improve conversions. Do not, however, test all the changes at once in a single A/B test if you want to know what exactly causes the improvement (if any) and to find the best combination (for example, which call to action works best with which image). In the example above, there are five page sections (elements) with two variations each. They make 32 possible combinations. If you run that as an A/B test, you’ll have no idea what made the difference.

4. Calculate estimated time to completion
It is possible to create a test that will take years to end. Say you have six elements and three variations per element, your test page gets 5,000 page views per day, your conversion rate is 4%, and you expect an improvement of 10%. That multivariate test would take 30 years to complete even if you include all visitors in your test. Use a duration calculator to find out whether the test you’re planning to run is actually reasonable; Google’s calculator can be found here.

5. Assess risks
While you may think of testing as gambling, rest assured that the odds are on your side. What you risk is a smaller amount of conversions in the short term, but what you can gain is a lot of conversions in the long run. Even when your hypothesis turns out to be wrong, you’ll learn something: how to avoid making costly mistakes in the future. To minimize risks, you might wish to expose a smaller share of your visitors to the test. But bear in mind that if fewer visitors are included, the test will take longer to complete.

6. Validate your implementation
Before your test goes live, make sure to test your implementation. Things that could skew your results include the prevalence and accuracy of test scripts (for instance, ensure that the goal script is implemented only on the actual goal page) and the loss of referral data. Some tools, Google Website Optimizer included, automatically validate the test for you. Do not blindly trust this because it is possible for your implementation to be flawed regardless of what the automatic validation says.

7. Avoid conflicting tests
If you set up two or more simultaneous tests sharing the same goal, you risk ending up with inaccurate results. Say you want to increase conversions and are running one test to find out which headline works best and a second test to find out which image works best. The two tests won’t share data, so you won’t know what combination of image and headline a converting visitor has been exposed to. If you’re running several tests, make sure that they won’t add noise and uncertainty to each other.

8. Look out for side effects
Wanting improvement is good. Being too eager to achieve results can, however, inadvertently lead to making costly mistakes—ones that are difficult to spot, too! While changes may increase conversions for one goal, they may decrease conversions for another. Typically a test is set up for only one goal. If you’re running an A/B test with Google Website Optimizer, use Google Analytics and custom variables to “tag” visitors. Then you’ll be able to find out how other goals and general visitor behavior are impacted by your test and its different variations. (Read more about the use of custom variables.) It’s also possible to include the goal script for Google Website Optimizer on multiple goal pages. But if you do that, both goals will be summed up together, and you won’t know the performance of each individual goal.

9. Try out segmented testing
The standard way of running a test assumes that one size fits all. While running any test is better than not testing at all, it is possible that one alternative/combination is better for direct traffic and another is better for traffic from Google AdWords (just an example). Therefore viewing the results for different segments will give you deeper insights. You can use custom variables to keep track of variations in Google Analytics and advanced segments to see how the different variations performed for different segments. You can also have a look at page 25 in The Techie Guide for Google Website Optimizer or find a tool more suitable for segmented testing at www.whichmvt.com.

10. Challenge the winner
So you’ve reached a statistically valid result of a test and announced the winner. Are you done now? No! Testing is not a one-time thing. It should be part of your process for continuous improvement. It’s not only possible, but likely, that there is a variation that will perform even better than the winner. That’s why a winner should be challenged from time to time. After all, Carl Lewis is no longer the fastest 100-meter dash runner, is he? At some point, there will always be a new winner.

View my presentation, “Top 10 Terrific Testing Tips”, from eMetrics Stockholm (PDF)


Many editors feel that once content has been created and uploaded, their job is done. Wrong! The next step is to analyze how well that content is doing its job. Using a web analytics tool, you can measure its effectiveness. It is possible to measure how well the content is performing even for purely informational websites, including intranets, that are not supported by ad or product sales.

If you don’t measure, you can’t analyze. If you don’t analyze, you can’t improve. Get Google Analytics if you’re not already using a tool. It’s free, so there is no excuse not to use it. And, yes, it works for intranets, too.

One of the first things you need to do—a step that is easy to miss—is to make sure you filter out writers and editors from your stats. Otherwise you’ll get inaccurate data.

You are not allowed to store personally identifiable information (such as social security numbers or names) in Google Analytics. You may, however, store data that can group visitors together in different clusters. For intranets that may be things as divisions and departments. For websites it could be consumers and businesses. By using custom variables to group visitors together, you’ll be able to analyze how different groups of visitors behave.

Keep in mind that file downloads are not automatically tracked by Google Analytics. If you want to track, say, PDFS, you need to trigger a virtual page view when a visitor is clicking on links to them.

Six measurements to maximize the impact of your content
Here are a few key performance indicators (KPIs) you can track to see how well your information is performing. There are many different things that can be measured—which KPIs are best to track varies from company to company, site to site. If you feel uncertain, talk to a web analytics consultant.

Too many pages and don’t know where to start? Prioritize which pages to optimize by looking at such things as bounce rate, read rate, $-index (monetary or goal value contribution) and satisfaction. Also check page view volume (tells you something about impact) and number of entrances (tells you how relevant the bounce rate is). Some of those metrics require additional configuration and JavaScript code on your website.

1. Readings per writer and writer impact
Find out how many times articles by a particular writer are being read, the number of conversions that happened during visits where the writer’s content was read, how long those visits lasted (best measured in intervals, use the Length of Visit report under Visitor Loyalty), and more. This is done by segmenting and creating custom reports based on a custom variable, as described below.

How to measure it: You need to edit the tracking script on your website that is used by Google Analytics. Trigger a page-level custom variable after a certain number of seconds on a page. Base the timer on a reasonable amount of time it takes to read an article. Add the name of the writer to a custom variable and the URL + “(read)” in _trackPageview(). This will result in the writer’s name being tied to the page and the page being marked as read. In “Top Content” you’ll have two entries for article1 looking like this: “/article1/index.php” and “/article1/index.php (read)”. You can filter the Top Content report by “(read)” to see how many times different articles have been read. If you only want to look at articles by a specific writer, use an advanced segment based on the custom variable containing the writer’s name.

Keep in mind that triggering an extra page view, as will be done above, will affect other metrics such as number of page views, page views/visit and bounce rate.

2. Page impact
If you have configured goals, you can find out how frequently visits, during which a certain page was viewed, led to goal completions and what the total goal value was for those visits. How well have those visits performed compared to other visits?

How to measure it: Create an advanced segment including only visits during which a specific page, or a certain group of pages, has been viewed.

3. Satisfaction
Measuring visitor satisfaction per writer and page adds an important extra dimension to your analysis. Imagine if you could find out how satisfied readers of specific writers are? Well, you can!

How to measure it: There are two ways. You can incorporate the possibility to give a rating for an individual page, or you can trigger a survey after the user has done certain actions, spent a certain amount of time on the site, or left the site. The key is to store the grade in Google Analytics. By combining behavioral and attitudinal data in Google Analytics, you’ll be able to see how frequently certain pages, or writers, have been involved in high and low satisfaction visits. Survey data can be stored as custom variables or virtual page views depending on what works best for you.

4. Time spent writing vs. reading
This measurement can tell you if there are pages that readers spend less time reading than the writers and editors spend creating. It is particularly valuable for intranets or websites with low traffic. If you spend a lot of time on an article and few people read it you’ve either failed at marketing the article or it’s simply not appealing to readers. Besides learning what content works best, this KPI will let you know how well writers are spending their time.

How to measure it: This requires a CMS hack.

Caveat: It will paint the true picture only if all writing is done directly in the CMS.

5. Unread pages
Pages need to generate at least one page view to be included in Google Analytics. Pages with no views won’t be listed. If you do not track unread pages, you won’t be able to accurately measure the average amount of page views per article or the average amount of readings per article. Both of those metrics are valuable to look at to evaluate content.

How to measure it: To find out which pages did not generate a single page view or reading, you need to match a list of pages from your CMS with page view data from Google Analytics. I recommend using Excellent Analytics, a free plug-in for Microsoft Excel, when combining data from Google Analytics with data from other systems. You can download it for free from: http://excellentanalytics.com/.

6. Cost per reading
Some organizations have goals stating that writers need to produce a certain amount of articles per week. I believe that it is more important to measure the impact of a writer’s articles.

How to measure it: Cost per reading = writer cost/number of times the writer’s articles have been read.

Better articles should generate more readings. Writers should strive to achieve a low cost per reading and a high satisfaction score.

---

Note: I originally wrote this article for Website Magazine in March. As a favor to them they got to use it exclusively until now.