It’s only natural.  When you look at the results of your testing, you always want to know why.

A/B TestingSometimes the test confirms your suspicions and you confidently boast your rationale.  In other cases, the test surprises you and you start looking for other reasons why the test went the way it did.

Whether you’re testing a banner ad, a landing page, an email or direct mail, we all want to understand why one version beat another.

It’s so easy and so satisfying to jump to a conclusion.

But too often when I see test results, there is no conclusion that could be made from the test – other than the fact that one version beat another.

And the reason is simple.  There are too many variables in each test version … so when one version wins, who can say which of those variables produced the winning results?   If you don’t know which variable drove the results, how can you then apply your results to future campaigns?

With that said, I applaud anyone who does any form of testing at all.  Although I believe testing is being used more often than ever before, it is still only a minority of marketers who is actually doing it.

And while their testing is not perfect, they are learning something about their marketing.

Let me give you two examples.

A/B TestingI recently sat in on the 30-minute web clinic by Marketing Experiments, a company dedicated to conducting testing and other marketing experiments.  You should check them out.  These video web clinics are very informative.

In this particularly web clinic, they were showing the results of an email test.   You can watch the video here.

One version – which was the control – was a traditional ad-format email.  You can see it at the top of this screen capture. This version looks a lot like a display  ad.  The test version was a letter-styled email that featured a shorter, simpler message with one image and one call-to-action button.

The results were clear-cut.  The letter style email beat out the display style email by a 3 to 1 margin.  The letter style email generate a conversion rate (not a click through rate of .12% while the display email only generated a conversion of .04%.ME - email test2

But what wasn’t clear-cut was why.  There were too many differences beyond the display-style vs. letter-style question.

I have my own reasons why the letter styled version won.  The letter version had a much stronger headline and a cleaner, easier-to-read design.  How would the results have changed if we simply applied that headline and design to the display version?

If you watch the video, you may have your own reasons.  Frankly, I was rooting for the letter-styled email because it has always been my preference.  For me, it seems to create more of a one-to-one connection.

But opinions only go so far in testing.

My second example is Which Test Won

I recently attend a presentation from Which Test Won, a terrific website and newsletter that shares marketing tests from contributing marketers.  In this presentation, about a dozen examples of landing pages were shown – and in just about every one of them, it was impossible to draw conclusions from the tests.

Don’t get me wrong.  It was fun evening with a lot of interesting tests and no shortage of opinion from all of us as to why.

But again, pretty much every test included multiple variables that make a pure testing impossible.


The lesson with all this is if you want to know which call-to-action text is most effective, you don’t change your button colors too.  If you want test a headline, you don’t change the body copy too.  If you want to test layouts, you don’t change the photos too.

You keep everything else exactly the same so you can isolate the single item you want to test.

And then you can draw a conclusion – with confidence.


Again, I highly recommend both of these websites if you are interested in testing and measurable marketing.  You can join their lists for free and stay informed of new content.


Written by Bob McCarthy

Download my FREE report, Step by Step Lead Generation & Lead Nurturing.

Join my list and get my newsletter.


Bob McCarthy is a direct response copywriter and lead generation specialist.

Download his Freelance Copywriting Information Kit.

copywriting kit graphic

About Bob McCarthy

Bob McCarthy is a direct response consultant and copywriter with a focus on direct mail, email and digital marketing. Bob works with B2B, B2C and Non-Profit clients. You can download his free ebook, "Making Snail Mail Work: 13 Lessons in Direct Mail Strategy."