Direct Mail Testing – A/B split and multi-variable testing

Direct mail testing is a critical, but often overlooked, component of a direct mail campaign.  More than anything, direct mail testing helps you learn about your marketing.  McCarthy & King Marketing provides strategically focused direct mail marketing programs for businesses and non-profits. 


Why test?

For those who are committed to direct mail marketing, testing is a high priority.

But for those who dabble in it or just want to experiment with direct mail, testing is a bit of an annoyance.

What the serious and long-term direct marketers understand is that testing is not a one-time activity that you do before your campaign begins.  It’s an ongoing process that helps you answer questions about every aspect of your marketing.

Sometimes testing confirms what you already knew.  Other times it can surprise you with results that may have gone against your intuition and best judgment.

The testing mindset says never be satisfied.  Always look for ways to improve.  Don’t rely entirely on your personal experiences.  Let the numbers do the talking.  You may be surprised.

A/B Split Tests

The most common type of testing is what is called A/B split testing.

A/B splits involves testing the exact same direct mail package except for one element in which two versions are created – an A version and a B version.

By keeping all other elements exactly the same, you can determine the relative effectiveness of the two test versions.

This may involve testing two lists, two offers, two formats, two envelope headlines, etc.

Testing two elements is the minimum, but you could test three elements at the same time.

If, for example, you are testing three lists, you would have an A/B/C split.


Multi-Variable (Multi-Variate) Tests

As the name suggests, multi-variable testing involves testing multiple elements at one time.

For example, you might test three lists and two offers and maybe two formats all within the same direct mail campaign.

While this may seem complicated, the biggest obstacle is cost because you need to create separate test cells for each version in the test (with possibly 5,000-10,000 names for each test cell).

Multi-variate testing is most often used for beginning programs when there are so many questions.  Eventually as your program matures, you will have some sense of what is working – and your testing may be limited to more focused tests.


How to test

Think of your direct mail campaign as a scientific laboratory.

When scientists tests their experiments, they create a “control” sample and then they create a variable to be tested against the control.  In this test, both samples are exactly the same except for one element (the variable).  When the results come back, they can see whether that variable was able to improve their results.

Direct mail testing works the same way.  By changing just one element in each test cell, you can see how that element affects response.


What to test

Testing any element in your direct mail program can be useful, but some elements are more useful than others.

The most important elements to test are the list and the offer – and should be your priority.

That’s because the list and the offer have the greatest impact on your results.

The list is the easiest element to test because you do it at virtually no additional cost.    Imagine if you were about the send out a 20,000 piece mailing.  You could send 20,000 mailers to the same list or your could send them to four lists of 5,000 each.  The cost of sending 20,000 piece is pretty much the same whether you use one list or four.

Contrast that with testing four different offers where you would have to create four separate mailers – which would cost you significantly extra.


How much to test

Testing will give you some numbers to examine.  The only question is how much confidence do you have in those numbers?

This is determined by the quantity you mail. The larger the quantity, the higher your confidence will be.

If you want to get sophisticated about this, you could learn about statistical probability, margin of error and confidence levels.  But this difficult material that few people understand.

Another approach (albeit less accurate than above) is to follow this simple calculation:

A reasonable mailing quantity is whatever it takes to produce 100 responses.   So for example:

  • a 2% response rate would require a 5,000-quantity test cell
  • a 1% response rate would require a 10,000-quantity test cell
  • a 0.5% response rate would require a 20,000-quantity test cell
  • a 0.25% response rate would require a 40,000-quantity test cell

These quantities may be way beyond your budget.  How will you test if you can’t afford these quantities?

Here’s how – test smaller quantities.  Test 2,500 or even 1,000.  It will tell you something even if the confidence level isn’t as high.  As long as you recognize the confidence level is low, you can continue with the same tests over multiple mailings to confirm your findings.


When to test

Contrary to public opinion, testing is not a preliminary step before you begin a campaign.  It’s an ongoing activity designed to continually improve your results.

When you’re just starting out, you will need to do more testing because you have no track record.  You have all sorts of questions (and no answers) about which lists, which offers, which formats or which message will work best.

But once your program matures and you have achieved some success, you will likely mail larger quantities to your “control” and smaller quantities to your test cells.


What to do with the results

When you are just starting out, you are looking for some direction, you are trying to answer some questions.  Your tests should help you with that.

If you have multiple test cells, you are likely to find some winners, some losers and some in between.

This is when you need to start making some decisions.

If there are no clear cut winners, you may want to continue testing the better options to see if a larger quantity will reveal a preference.

If you were testing lists, you need to find out whether there are more name from the same winning test samples. This is known as a “roll out” quantity.  Sometimes, you test a list and find out that the test amount was all there was.  If that’s the case (and you should know this before you test), it may have been better not to include that list in your test.

WARNING:  Many list companies have what is know as their “hot-line names” that you can rent.  These are typically the most recent responders to the list owner’s offerings.  While these hot-line lists may be a good source of new business for you, it’s not a good test sample because it doesn’t accurately reflect the rest of the list.


Get started or learn more

So what’s your next step?  If you’re close to making a move, we’d love to talk.  Call Bob McCarthy at 508-473-8643 or send him an email at

If you’re not there yet, let’s stay in touch.  Please join our mailing list and get all of our updates and articles.

Of course, if you have any questions, please don’t hesitate to call.

If you’d like to discuss an upcoming campaign – or if you’re looking for ways to improve an existing mailing program – call us and let’s schedule an introductory call.