How to Improve Performance with A/B Testing

A/B Testing process refers to testing two alternative approaches against each other to find the best of the two.  It is retained and, in turn, compared to a third alternative.  The best of these is retained and the process repeated.

Very often, we have no real idea of what will appeal to customers so this, real world, testing avoids your own unconscious biases impacting on the success of your activity when you only test one option.  Yellow Belt

Always be Testing

The A/B testing technique provides 2 or more options.  It tests the appeal of each with a positive action taken by the people seeing the paid advertising.

Example:

Test the popularity of the Product A against the Product B.

If more people click on the Product A, and go to the website discussing the product, than click on the Product B, one can assume that A is a better “Product” than B.

Test this assumption even more rigorously with the statistical techniques of “Significance”.

You can test if Product A is significantly better than the Product B.

Introduce other examples of the “Product” and test. Continually drop out the weakest and keep the strongest.

When you do have a clear cut, statistically “Significantly” “Product”, you can be confident that it is customer-friendly.

An interesting by-product of this is to see how many people are attracted to your “Product”. 

• This process may have identified the best example of your “Product”, but you might find that the click through traffic is poor.
• This indicates that, in fact, you have a problem with your actual “Product”.
• This may trigger you to re-think the design of this “Product”. 
• Especially if your potential clientele is too small to achieve the growth or profit goals that you have.

When testing the “Product”, you need a purpose built single page on your website. This is your “Landing Page”.

This page is rich in the terms that you have used in the “Product” advertisement.  Rich in both its Title Tag and Meta Tags.

Further reading on this topic at the article: Website Design

These Landing Pages should be in the menu structure of your website.
Place the link to the Landing Pages at the bottom of the page where the search engine crawling will find them.  Customers may not bother to go there, lessening confusion.

How long do you test for?

• The first round of A/B Testing is likely to quickly demonstrate the weakest of your alternative “Product”. 
• Having narrowed the range down, you may want to allow the test ads to run for a reasonable amount of time.
• This will avoid any seasonal difference.

As an obvious case:

• If you had a “Product” that said “Always deliver in time for Christmas”.  That would score very well in the Christmas season but not in February.
• It is a good idea to try to remove seasonality, where possible.  With such things as holiday traffic it is impossible to remove seasonality altogether.

It is up to you to decide how long you run the advertisements. 

• Best practice suggests that you can drop the weak options in as little as 30 days. 
• When deciding your best option, you may want to run for around 90 days.

What To A/B Test

Above, we indicated there are a number of different criteria that you can use to differentiate yourself in the mind of the customer.

You may be immediately tempted to go with the one that works best for you (say lowest price).
But, you may be failing to find that there is even a more powerful one that you didn’t bother to test.

Therefore, divide testing into 2 major steps:

1. Attempt to find candidate descriptions of your “Product” in more than one of the several criteria given above.

2. For each of the different types of differentiation, you should have at least 2 advertisements.

• This helps to ensure that it is not a poor advertisement that causes, what might have otherwise been your best “Product”, not to be chosen very often with your paid ad traffic.

• Having at least 2 advertisements for each differentiation, may mean around 6 to 12 different paid advertisements running at the outset of the test cycle. 
  • You will be confident that you have bracketed the possible differentiators and given them a fair chance to stand out in your testing.

• The poor differentiators will show up quite quickly.  None of their several advertisements will work.

• Some advertisements for some differentiators work well, other advertisements do not. 
  • Revisit that differentiator with new angles of attack on the advertisements.
    That way you can, over time, decide whether it is the advertisement that is weak or the differentiator.
    The last thing you want to do is remove a good differentiator from consideration simply because you couldn’t think up a good advertisement at the start.

• This sounds like identifying the best “Product”, by a process of testing, is a cumbersome, long and tiresome process.
  • Think of what is the cost of missing out on your ideal “Product”.  Particularly if it is because you couldn’t be bothered to test it at the start.

Testing the Strength of Buyer Sentiment

Depending on the nature of your business, and the level of comfort you want when choosing your best “Product”, you may want to consider the following:

• Counting click throughs:
  • This is the number of people who have clicked on your advertisement containing the test “Product”.
    The number of click throughs can be impacted by various things.
    You want to know the “CTR” (Click Through Rate).  It tells you the percentage of people who have clicked on your advertisement.

• The percentage is more important than the actual number of people, in many ways.
  • You can get more click throughs by paying for more advertising and appearing on more pages.
    Therefore, the total count may be hiding the fact that you have bought the total count by paying more money.
    You want to pay as little as possible to get clicks so that your future marketing with this “Product” is as cheap as possible.

• You want to test the level of commitment and interest.
  • Having people click on your advertisement may simply indicate that they have been attracted by the phraseology about your “Product”. 
    This is, in its own right, a good thing but what you really want is people who are committed to buy.
    You want a Lead Qualification device to be sure that they are sufficiently interested to give you, for example, their email address.
    Today, if people give you an email address, it is a powerful indication that they are attracted to what you have to offer.  Either for now or in the future.

Further reading on Lead Qualification is in the article: Sales Funnel