The Panel Test: Great Bang for the Buck
| Category: E-mail Marketing | Date: 2002-06-06 |
Testing multiple variables within a single email campaign can get a bit tricky. One way to do it is with the all-important and extremely valuable "panel test." It requires pretty aggressive numbers for deployment, yet can be an excellent way to get to know the strongest (and weakest) responders and/or variables within a mailing. And as far as applications for reading and applying results go, this method of testing can get you the most bang for the buck.
Essentially, a panel test comprises individual groups of email lists, each of which consists of like-minded people. Each group represents a different variable that youd like to test and is flagged appropriately. When it comes time for deployment, all groups are sent at the same time. (This is critical.) And when the campaign is complete, final results are determined based on reading response at both the list and the group level.
Lets take a closer look. Say you have an online party supply store and have a house list of 200,000 previous buyers and/or newsletter subscribers. You consistently send them, en masse, a promotional newsletter every two weeks in an attempt to get them back to your site to buy. Response is okay, but it could be better, in your opinion. Your goal? To find the cream of the crop within your audience and which demographics and segments will produce the best results when matched across your artillery of offers, products, subject lines and creative, in order to enhance your results.
First order of business: Split your list. To avoid too much confusion and for purposes of this article, lets just say that you have the ability and want to split your list according to 1) gender, 2) whether the previous buyers made purchases within the last 30 days, 60 days or six months. Simple enough.
As far as the offer goes, you want to pit your tried-and-true newsletter (which contains content AND promotes products) against a straight product offer that has two different versions. One version promotes one product only while the other promotes multiple products. Added to that is a test of two different subject lines, along with two different pieces of copy. Whew!
So to clarify, and because Im a visually oriented person, heres a grid to demonstrate the various panels and their contents. (Ill address the blank boxes in a moment.)
Panel 1 -
Newsletter
(Control)
Panel 2 -
One
Product Panel 3 -
Multi-
Products Panel 4 -
Subject
Line 1 Panel 5 -
Subject
Line 2
Panel 6 -
Copy A
Panel 7 -
Copy B
MALE MALE MALE MALE MALE
FEMALE FEMALE FEMALE FEMALE FEMALE FEMALE FEMALE
LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS
LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS
LAST 6
MO LAST 6
MO LAST 6
MO LAST 6
MO LAST 6
MO
The quantity of each subsection or segment (i.e., male, female, etc.) should be equal within each panel and across all panels. And you dont necessarily have to email your entire list. Remember, this is a test. As long as you have taken an equal number of random names across each segment — representing similarly minded people, presumably — that is the most important thing. To be statistically significant (and to ensure the validity of the campaign), most email marketers will split the list into groups of at least 3,000 to 5,000 people per segment.
One rewarding thing to note about panel testing is that you can lower your risk a bit by reducing the number of people in your mailing that you might assume are lower-than-average responders, such as those who havent bought in a while. In this case, the assumption was made that both "males" and older (last six months) buyers were higher risk. So in two of the panels — Panel 5 and Panel 7 — those segments were not emailed the corresponding version.
However, results could still be determined based on the relative strength of the counterparts. In other words, Panel 4 (Subject Line 1), which is Panel 5s counterpart (meaning everything else is the same except for the subject line), will get results for all segments. Panel 5 can also learn some things about its own components based on Panel 4s results. So say the click-through rates for each segment of those two panels shake out as follows:
Panel 4 -
Subject Line 1 CTR Panel 5 -
Subject Line 2 CTR
MALE 3.6% Didnt email
FEMALE 7.9% FEMALE 8.3%
LAST 30 DAYS 11.2% LAST 30 DAYS 11.1%
LAST 60 DAYS 3.6% LAST 60 DAYS 4.0%
LAST 6 MO 1.2% Didnt email
As you can see, the results clearly indicate that, on an overall basis, Panel 5 – Subject Line 2 is stronger than Panel 4. The slight difference in the "30 days" lists results is not significant enough to cause concern. And the response rates of the segments that were a part of both panels indicate that the weaker segments will be relatively as strong as their counterparts. Meaning the click-through for "males" and older (last six months) buyers would have been, in all likelihood, stronger in Panel 5 had they been mailed.
When all is said and done, your final results will be based on both the overall results of each panels weighted average, along with the results of the individual lists within each panel. Sometimes a panel will be an overall winner, with one or two lists being clear-cut losers. When that happens, its simply a matter of doing a little tweaking here, a little fine-tuning there. Before you know it, youll have optimized your email campaigns. And isnt optimization what testing is all about?
About the Author
Kim MacPherson is President and founder of Inbox Interactive (formerly known as Selling By Design), a D.C. area-based online direct marketing agency specializing in email promotional copywriting, HTML design, and planning. She is also a consultant and frequent speaker on the topic of email marketing and is the author of the upcoming book "Email Marketing 101" to be published by Dearborn. You can email Kim at inboxinteractive.com.
kim@inboxinteractive.com
Essentially, a panel test comprises individual groups of email lists, each of which consists of like-minded people. Each group represents a different variable that youd like to test and is flagged appropriately. When it comes time for deployment, all groups are sent at the same time. (This is critical.) And when the campaign is complete, final results are determined based on reading response at both the list and the group level.
Lets take a closer look. Say you have an online party supply store and have a house list of 200,000 previous buyers and/or newsletter subscribers. You consistently send them, en masse, a promotional newsletter every two weeks in an attempt to get them back to your site to buy. Response is okay, but it could be better, in your opinion. Your goal? To find the cream of the crop within your audience and which demographics and segments will produce the best results when matched across your artillery of offers, products, subject lines and creative, in order to enhance your results.
First order of business: Split your list. To avoid too much confusion and for purposes of this article, lets just say that you have the ability and want to split your list according to 1) gender, 2) whether the previous buyers made purchases within the last 30 days, 60 days or six months. Simple enough.
As far as the offer goes, you want to pit your tried-and-true newsletter (which contains content AND promotes products) against a straight product offer that has two different versions. One version promotes one product only while the other promotes multiple products. Added to that is a test of two different subject lines, along with two different pieces of copy. Whew!
So to clarify, and because Im a visually oriented person, heres a grid to demonstrate the various panels and their contents. (Ill address the blank boxes in a moment.)
Panel 1 -
Newsletter
(Control)
Panel 2 -
One
Product Panel 3 -
Multi-
Products Panel 4 -
Subject
Line 1 Panel 5 -
Subject
Line 2
Panel 6 -
Copy A
Panel 7 -
Copy B
MALE MALE MALE MALE MALE
FEMALE FEMALE FEMALE FEMALE FEMALE FEMALE FEMALE
LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS LAST 30
DAYS
LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS LAST 60
DAYS
LAST 6
MO LAST 6
MO LAST 6
MO LAST 6
MO LAST 6
MO
The quantity of each subsection or segment (i.e., male, female, etc.) should be equal within each panel and across all panels. And you dont necessarily have to email your entire list. Remember, this is a test. As long as you have taken an equal number of random names across each segment — representing similarly minded people, presumably — that is the most important thing. To be statistically significant (and to ensure the validity of the campaign), most email marketers will split the list into groups of at least 3,000 to 5,000 people per segment.
One rewarding thing to note about panel testing is that you can lower your risk a bit by reducing the number of people in your mailing that you might assume are lower-than-average responders, such as those who havent bought in a while. In this case, the assumption was made that both "males" and older (last six months) buyers were higher risk. So in two of the panels — Panel 5 and Panel 7 — those segments were not emailed the corresponding version.
However, results could still be determined based on the relative strength of the counterparts. In other words, Panel 4 (Subject Line 1), which is Panel 5s counterpart (meaning everything else is the same except for the subject line), will get results for all segments. Panel 5 can also learn some things about its own components based on Panel 4s results. So say the click-through rates for each segment of those two panels shake out as follows:
Panel 4 -
Subject Line 1 CTR Panel 5 -
Subject Line 2 CTR
MALE 3.6% Didnt email
FEMALE 7.9% FEMALE 8.3%
LAST 30 DAYS 11.2% LAST 30 DAYS 11.1%
LAST 60 DAYS 3.6% LAST 60 DAYS 4.0%
LAST 6 MO 1.2% Didnt email
As you can see, the results clearly indicate that, on an overall basis, Panel 5 – Subject Line 2 is stronger than Panel 4. The slight difference in the "30 days" lists results is not significant enough to cause concern. And the response rates of the segments that were a part of both panels indicate that the weaker segments will be relatively as strong as their counterparts. Meaning the click-through for "males" and older (last six months) buyers would have been, in all likelihood, stronger in Panel 5 had they been mailed.
When all is said and done, your final results will be based on both the overall results of each panels weighted average, along with the results of the individual lists within each panel. Sometimes a panel will be an overall winner, with one or two lists being clear-cut losers. When that happens, its simply a matter of doing a little tweaking here, a little fine-tuning there. Before you know it, youll have optimized your email campaigns. And isnt optimization what testing is all about?
About the Author
Kim MacPherson is President and founder of Inbox Interactive (formerly known as Selling By Design), a D.C. area-based online direct marketing agency specializing in email promotional copywriting, HTML design, and planning. She is also a consultant and frequent speaker on the topic of email marketing and is the author of the upcoming book "Email Marketing 101" to be published by Dearborn. You can email Kim at inboxinteractive.com.
kim@inboxinteractive.com
Copyright © 2005-2006 Powered by Custom PHP Programming