It’s only natural. When you look at the results of your testing, you always want to know why.
Sometimes the test confirms your suspicions and you confidently boast your rationale. In other cases, the test surprises you and you start looking for other reasons why the test went the way it did.
Whether you’re testing a banner ad, a landing page, an email or direct mail, we all want to understand why one version beat another.
It’s so easy and so satisfying to jump to a conclusion.
But too often when I see test results, there is no conclusion that could be made from the test – other than the fact that one version beat another.
And the reason is simple. There are too many variables in each test version … so when one version wins, who can say which of those variables produced the winning results? If you don’t know which variable drove the results, how can you then apply your results to future campaigns?
With that said, I applaud anyone who does any form of testing at all. Although I believe testing is being used more often than ever before, it is still only a minority of marketers who is actually doing it.
And while their testing is not perfect, they are learning something about their marketing.
Let me give you two examples.
I recently sat in on the 30-minute web clinic by Marketing Experiments, a company dedicated to conducting testing and other marketing experiments. You should check them out. These video web clinics are very informative.
In this particularly web clinic, they were showing the results of an email test. You can watch the video here.
One version – which was the control – was a traditional ad-format email. You can see it at the top of this screen capture. This version looks a lot like a display ad. The test version was a letter-styled email that featured a shorter, simpler message with one image and one call-to-action button.
The results were clear-cut. The letter style email beat out the display style email by a 3 to 1 margin. The letter style email generate a conversion rate (not a click through rate of .12% while the display email only generated a conversion of .04%.
But what wasn’t clear-cut was why. There were too many differences beyond the display-style vs. letter-style question.
I have my own reasons why the letter styled version won. The letter version had a much stronger headline and a cleaner, easier-to-read design. How would the results have changed if we simply applied that headline and design to the display version?
If you watch the video, you may have your own reasons. Frankly, I was rooting for the letter-styled email because it has always been my preference. For me, it seems to create more of a one-to-one connection.
But opinions only go so far in testing.
My second example is Which Test Won
I recently attend a presentation from Which Test Won, a terrific website and newsletter that shares marketing tests from contributing marketers. In this presentation, about a dozen examples of landing pages were shown – and in just about every one of them, it was impossible to draw conclusions from the tests.
Don’t get me wrong. It was fun evening with a lot of interesting tests and no shortage of opinion from all of us as to why.
But again, pretty much every test included multiple variables that make a pure testing impossible.
Takeaway
The lesson with all this is if you want to know which call-to-action text is most effective, you don’t change your button colors too. If you want test a headline, you don’t change the body copy too. If you want to test layouts, you don’t change the photos too.
You keep everything else exactly the same so you can isolate the single item you want to test.
And then you can draw a conclusion – with confidence.
———–
Again, I highly recommend both of these websites if you are interested in testing and measurable marketing. You can join their lists for free and stay informed of new content.
—————
Written by Bob McCarthy
Download my FREE report, Step by Step Lead Generation & Lead Nurturing.
Join my list and get my newsletter.
—————
Bob McCarthy is a direct response copywriter and lead generation specialist.
Download his Freelance Copywriting Information Kit.
- A High-Wire Act for Today’s Marketers - January 28, 2021
- Is it content – or just copy? - May 26, 2020
- How you can – and why you should – use your website to teach - May 12, 2020
- How to Generate Qualified Leads with Lead Surveys - March 1, 2020
- Gaps in your Email Schedule – and how to avoid them - September 4, 2019
- The Dizzying World of Letter Mail - July 22, 2019
- Do you have an Information Kit? - June 25, 2019
- Direct Mail and Those Pesky Millennials - November 8, 2017
- Landing Pages: Do you need a squeeze page or sales page – or both? - May 31, 2017
- Do you have the patience for content and inbound marketing? - May 6, 2017