A/B Testing Gone Awry

Today, I want to respond to a very interesting video posted by YK Sugishita about A/B Testing. Before reading the rest of this blog post, please enjoy this 4 1/2 minute video.

At JC Digital, we are big fans of A/B Testing. But as the video points out, if you don’t have a clear strategy of where you want to be, it can quickly devolve into a cluttered mess of testing inane things.

The problem is you depend on employees of your company, or worse, executives for opinions on what to test next. Sure, employees should have ideas and opinions on how to improve your site. Nothing wrong with that. And dealing with the “Highest Paid Person’s Opinion” (HiPPO) needs be handled gracefully. But as employees, we are way too familiar with our product. We have been drinking our own Koolaid for way too long.

If you are not careful, you end up testing the wrong things to that don’t make up much impact on the overall goal. You waste your precious time. In a humorous way, the video points this out very well.

So after watching this you might want to give up A/B Testing completely….. Don’t do it.

A more systematic approach to A/B testing is necessary.

We recommend these steps:

1. First, a full analysis should be done of the entire funnel for attracting customers. That analysis should examine every step, to see what steps went wrong. Analytics, and other tools should be employed to find the weakest points.
2. If a broken stage (or page) in the funnel is discovered, then more local page analysis should be conducted as well. You can often get a clue by looking at heat maps, scroll maps to determine where users are falling off.
3. We also recommend doing actual user testing, via usertesting.com or some similar service. Nothing beats watching actual users trying to navigate through your intended funnel.
4. Finally, we are big fans of online popup surveys. There are a couple of techniques that can be employed here. One technique is to popup a 1-question survey to ask there users “why they won’t buy today?”. Or ask you customer who bought, “what was the 1 hesitation you had before you bought?”.

So now armed with actual meaningful data, you can now make some intelligent decisions on how to conduct your A/B tests.

You might find one of the following issues.

• Not all the users concerns have been met, resistance is still high
• Your website has a trust issue
• Call-to-action button not prominent
• Not clear what the next step is, CTA not clear
• Unique benefit to the user not clear, or feel it does not apply to them

and so on.

These are the types of realistic things you can and should test with A/B testing.

The best part about steps 3 & 4, is you can get the user comments in their own words. The words they use can then be used in the copy (or text) of your A/B Test.

Now getting back to the video, there are certain cases where testing the size or the placement of a button might give you some lift in conversion rates. That might be helpful if you are the size of the Google Search or Amazon store, where millions of users view the page each day. For those companies that have already optimized that funnel so much, then go ahead and test the button size or color. For the rest of us, we have much bigger fish to catch.

In summary, it pays to be intelligent about what to choose next for your A/B test. It’s OK to listen to employees opinions. But way better is to systematically locate your pain points using analytics, then choose your next A/B test by aggregating the feedback of your users. And above all, as the video points out, it’s important to have a goal.

Happy A/B Testing.