Testing is the path to success in direct mail.
Generally, testing involves just two elements – testing one item against another … such as two different mailing lists, or two offers, or two formats or two messages or two of anything else.
This is known as an A/B split test.
You select the two items that you want to test, then mail half with the first item, and the other half with the second item. The key is to keep everything else in the direct campaign exactly the same. In that way, you know for sure that any difference in response is the result of the testable item.
You’re not limited to two items of course. If you want to test 3 lists for example, you split your mailing into three groups and conduct an A/B/C split test.
An A/B/C test would involve testing three of the same thing – that is, three lists, three offers, etc.
This is a fairly common test strategy especially with mailing lists. It would not be unusual to see a test of five to 10 (or more) mailing lists in one mailing.
But what if you want to test two different elements in your direct mail campaign?
What if, for example, you want to test two lists and two offers?
For that, you would need to set up a direct mail test matrix.
A test matrix allows you test multiple items at the same time. In the example below, you would split your mailing into four segments, or test cells – A1, A2, B1 and B2.
As with every test, all other elements of the direct mail campaign are exactly the same.
| Offer A | Offer B | |
| List 1 | A1 | B1 |
| List 2 | A2 | B2 |
With this test matrix, you will be able to see several results:
- which combination of list and offer performed the best of four test cells
- which of the two offers produce the better result
- which of the two lists produced the better result
You’re not limited to two offers or lists. Here’s an example of a test with seven lists and two offers.
| Offer A | Offer B | |
| List 1 | A1 | B1 |
| List 2 | A2 | B2 |
| List 3 | A3 | B3 |
| List 4 | A4 | B4 |
| List 5 | A5 | B5 |
| List 6 | A6 | B6 |
| List 7 | A7 | B7 |
But a test of this size can add up pretty quickly in terms of mailing quantities. A test cell is often 5,000 – 10,000 names (you can test smaller quantities but with less reliability) so a 14-cell test quantity will require a mailing of 70,000 – 140,000 pieces.
What if you can’t afford to mail that many pieces?
Here’s alternative approach.
| Offer A | Offer B | |
| List 1 | A1 | B1 |
| List 2 | A2 | |
| List 3 | A3 | |
| List 4 | A4 | |
| List 5 | A5 | |
| List 6 | A6 | B6 |
| List 7 | A7 |
Notice we removed all but two of the offer tests. This reduces your total test cells from 14 to 9.
At the same time, you are still able to test the relative effectiveness of seven different lists (using Offer A) and the relative effectiveness of your two offers with smaller quantities.
Why did we select B1 and B6? It’s a random selection. Unless one of the lists is the control (an established winner), it could have been any two cells. If you do have a control in place, you would want include the control list in the offer test because that list has a proven track record with you.
We want to use at least two test cells so that we could verify our results. Relying on just one test cell could be risky.
If, for example, Offer A beats Offer B in one test cell, but loses in the second test cell, we are left with some doubt. But if Offer A wins in both cells, we can feel confident in our results.
If it makes you feel any better, use a third test cell.
I have used the list and the offer in these examples in part because they are the most important elements to test. But you can create a direct mail test matrix for any element of your direct mail or email campaign.
- A High-Wire Act for Today’s Marketers - January 28, 2021
- Is it content – or just copy? - May 26, 2020
- How you can – and why you should – use your website to teach - May 12, 2020
- How to Generate Qualified Leads with Lead Surveys - March 1, 2020
- Gaps in your Email Schedule – and how to avoid them - September 4, 2019
- The Dizzying World of Letter Mail - July 22, 2019
- Do you have an Information Kit? - June 25, 2019
- Direct Mail and Those Pesky Millennials - November 8, 2017
- Landing Pages: Do you need a squeeze page or sales page – or both? - May 31, 2017
- Do you have the patience for content and inbound marketing? - May 6, 2017
Bob, well presented explanation of testing.
I have two questions:
1. Do you believe/use a “test control” panel for exact measurement size. (That is, a panel the same size as each test panel, but mailing the control offer to known lists)?
2. Do you agree with a “challenge” test, where only the product and price is the same as the control, but all other components and elements are tested at once? (The challenge allows for some real “new eyes breakthrough” ideas…)
I look forward to your reply…
Peter T. Britton
The Write Answers
Hi Peter
Thanks for the comments and questions.
I’m not sure I follow the first question. If you are asking whether you need to use the exact same quantities for each test cell, the answer is no. In many cases, the control mailing would have a much larger quantity than any single test cell.
But the test quantities need to be sufficient to have confidence in your results. Most catalog and mail-order companies (with order generation applications) will have minimum test cell of 10,000 names. Lead generation could probably get by with 5,000 names.
I will often test quantities even smaller because the budget or market size doesn’t give me any more names. I do this with the understanding that the confidence level will lower and the potential for deviation will be higher. But it’s better than no testing.
I don’t know first-hand if this is a common practice, but when a test cell does beat the control, I think it’s advisable for a mailer to re-test the same test cell with increasing quantities to verify the results before establishing the test cell as the new control.
As for “challenge” test, I am aware of this practice although I’ve never participated. (Most of my work is in the lead generation sector). The “challenge” test gives new writers and creative teams the opportunity to show their stuff and try to beat the existing control.
As you said, it gives the mailer the chance to see entirely new approaches for their promotions.
This only works with mature programs that have already tested out their lists and offers. Because the list and the offer are the two most important elements (by far), there is a lot of testing that needs to be done before you get to the creative.
But even for mature programs, the “challenge” test has some shortcomings.
When your test does find a new winning package, what made it the winner? Was the overall message, the envelope copy, the headline, the photos, the testimonials, the added insert, the personalization … I could go on but you get my point.
All you know is that this particular mailer beat the control mailer – which is good information, but should be viewed as a first step. The next step should be to fine tune your testing to start answering these questions.
I hope this helps.