Five Secrets of Successful Testing
| Category: Marketing Strategy | Date: 2001-03-09 |
It has been my experience that not every database marketing tactic is successful. There are the reported programs that have been costly failures. Others have been only marginally successful in their basic goals which hopefully have been measured in quantitative units, revenue or time.
How can marketers insure that a database marketing tactic will be a success? Testing can significantly raise the chances for success. But to be true predictors of the outcome of a proposed campaign, the tests have to be more than a hastily selected random sample of consumers.
A successful test is defined as: any test which provides reliable results that are projectable and extendable to the target audience.
The results of any campaign must eventually be measured in some analysis to determine whether it was the offer, the price, the packaging, the advertising, the persuasiveness of the direct marketing campaign or just social or weather events that affected the campaign's outcome.
Today, when surveys are made to determine the extent of markets for a particular product, the trend is toward coupling new measuring methods that include the utilization of statistics and computer models to predict behavior with the traditional applications from the multi-disciplinary sciences and math.
The results of these combined methods are used to help predict sales projections, the type of advertising to use, the allocation of sales people, the number and location of warehouses and the amount of product at the retail locations.
Unfortunately, two significant issues are often overlooked in the haste to get to the testing of the audience. These are:
1. Who is the correct audience(s)?
2. What is the correct size of the test to insure statistically reliable results.
To insure that these two critical questions are answered, once the customer profiling (based upon transaction behavior and household characteristic) is done, smart marketers should divide the file by segments. Only those segments that most match the product and the expected behavior should be used for testing.
Most marketers want to run to test the following:
1. Product, The List, The Offer
2. The size of the test cell(s)
However, there is a third issue to test that is equally important.
3. Was the correct target audience really selected before the list was used or purchased?
Due to conflicting objectives (cost vs. size), the list is sometimes not examined adequately to determine if it really is the specific target audience. When conducting a test, marketers want to use test samples that are as large as necessary to ensure scientific accuracy but as small as possible to keep the cost of the test to a minimum.
The normal marketing behavior is to select a list that provides a close approximation of the "general audience." However, it is vital to move away from the mass mailing and move closer to the targeted mailings which are relevant to the audiences or segments which most closely approximate the profile of the product.
Take this model for example:
Selecting the best audience takes the random selection and guesswork out of choosing which group will respond the best, because it matches the customer demographic, psychographic and behavior to the product or service being sold.
The Product:
Solve the business problem first. Be sure that your business objectives match the potential of the product to deliver adequate profit to support a reasonable roll-out, assuming that you have a successful test. A series of ROI scenarios will quickly determine where the test is warranted.
Even some very large companies have failed in this planning. One example is the breakfast food marketing campaign of a very large food company. The campaign offered a free bonus product when consumers purchased one product. However, marketing forgot that the average consumption of the breakfast cereal was 3.5 months. No testing was done and the campaign was offered to the entire database. The marketing manager is still looking for a job.
The Lists:
Most professionals agree that the list is the most important element in any promotional marketing activity. Since all tests are on a tight budget, how can marketers determine the ideal minimum size to satisfy the test objectives?
1. Select a number from your own database that is projectable if the test proves successful.
2. Test a sufficient number of lists. If you test only one list and the test fails, you won't know if it was due to the list or other factors.
3. If you are purchasing outside lists use a genuine, legitimately recognized broker or a company that is known for its ability to deliver target specific lists.
4. Send all the test mailings out standard mail first class, pre-sort in order to measure and recover and tabulate the deliverability of each list.
The Offer:
The third most important element in any test is the offer. How much should you offer in order to get prospects or consumers to respond? Remember the breakfast food company...Sometimes it take more than you expect to move the customer to action. Therefore, it's important to test various levels of the offer.
1. Analyze whether the offer is financially expandable to a larger audience. Trading margin dollars may or may not be in order to generate a response. Go back to solving the business plan.
2. Consider giving a higher level offer to the "best" customers or the highest prospects based upon their repeat purchase history. This is going back to who is the real target audience. Which target group is the one that offers the best value over a period of time (Lifetime Value).
3. Skip the evaluation of the blue envelope versus the pink envelope, or the clever saying versus the straight proposition on the outer envelope (OE). If you've done a good job of profiling your target audience you already know what turns them on.
Factors to consider when profile modeling is to be used:
1. Predictable Behavior: For a behavior to be predictable the database must include behavior history with the company.
2. The database must contain an adequate number of those customers. There should be at least 500 to 1,000 customers who have demonstrated the behavior you are trying to model for the list purchase.
As an example, let's assume that we are going to send a promotion to our customer base or to a group of prospects who resemble the profiles that have been determined. We could be using direct mail, fax, e-mail or outbound telemarketing. We have several approaches that we want to test. We will set up a group of customers, called "cells," that we will use to test each approach.
For each promoted cell, we will determine the response rate (RR). This rate, of course, is calculated by dividing the number of responders (buyers) by the number promoted (mailed or telephoned). Let's assume that we directed our promotion at 30,000 households and 600 of them accepted the offer. The response rate is: RR = (# respondents) / (# promoted) or 600 / 30,000 = 2.0 percent.
Is this an acceptable response rate? That is determined by whether we made or lost money. What is our return on investment (ROI)? That is calculated by dividing the net revenue by the cost of promotion. The net revenue is the gross revenue from sales, less the cost of the product, the cost of fulfillment, the returns and processing fees.
By calculating the total number of responders, the gross revenue generated from the sale less the variable and marketing costs will determine whether the program is a success. The following illustration demonstrates what happens to a proposed or actual effort.
Using the formula developed above, we can calculate -- in advance -- what level of response we would need from any promotion to just break even. Expressed as a formula, BE is when (net revenue) = (cost of promotion).
This formula tells us the break-even response rate for any combination of outgoing per-piece costs divided by any combination of average net revenue from successful sales. For example, if the per-piece cost of promotion is $5.99 and the net revenue from the average sale is $248.69, then the break-even rate is 1.59 percent.
How Many in Each Cell?
This returns us to the original question. How many consumers must we promote in each cell to be sure that the results we get accurately predict how other similar customers will react to a similar offer? A simple formula answers this question: Minimum cell size = 5.0 / break even. In other words, if the break-even rate is 1.59 percent (as above) then the formula is: 5.0 / .0159 = 314 customers who are expected to respond.
The object of testing is to get a statistically projectable and reliable number. We believe that by approaching the testing in segments that approximate the size of the database after segmentation delivers solid results. Each mailing offer and segment add to 10,000 and the response should be equal to the minimum break even of 159.
We've looked only at what it takes to break even. The next approach is to measure the significance between the break even and the real return. The difference is then assigned an index to describe how well each cell performed. For example, in "Offer One List A" the actual return was 120 and the difference is 40. Now compare to the other cells such as "Offer Two from List B" where the return was 100 or a difference of 20 less than "Offer One from List A." Let's assume that the average response was 30 above the break even. If 30 is the average, then "Offer One from List A" was indexed at 40/30 = 133 and "Offer Two from List B" was indexed at 67.
At first glance it appears that "Offer One from List A" was twice as effective. Now bring in the margin difference and compare the margin return. Margin on "Offer Two from List B" was 10 higher than that of "Offer One on List A." Which one do you think the person in charge is likely to select for the roll out?
There are many types of testing scenarios that have been tried and found successful. This is one. The statisticians out there may look at the database or mailing as homogeneous and would therefore prefer a different type of statistical approach.
I know many readers have experience with tests and controls, return on investment, break-even response rates and minimum test sizes. Their experience and logical framework may differ from the approach suggested here. If you have a different -- and perhaps better -- way of dealing with these concepts, I invite you to contact me so I can share your ideas in a future article.
Thanks to my friend Arthur Hughes for providing some framework for this article.
DBMarkets@aol.com
http://www.msdbm.com
How can marketers insure that a database marketing tactic will be a success? Testing can significantly raise the chances for success. But to be true predictors of the outcome of a proposed campaign, the tests have to be more than a hastily selected random sample of consumers.
A successful test is defined as: any test which provides reliable results that are projectable and extendable to the target audience.
The results of any campaign must eventually be measured in some analysis to determine whether it was the offer, the price, the packaging, the advertising, the persuasiveness of the direct marketing campaign or just social or weather events that affected the campaign's outcome.
Today, when surveys are made to determine the extent of markets for a particular product, the trend is toward coupling new measuring methods that include the utilization of statistics and computer models to predict behavior with the traditional applications from the multi-disciplinary sciences and math.
The results of these combined methods are used to help predict sales projections, the type of advertising to use, the allocation of sales people, the number and location of warehouses and the amount of product at the retail locations.
Unfortunately, two significant issues are often overlooked in the haste to get to the testing of the audience. These are:
1. Who is the correct audience(s)?
2. What is the correct size of the test to insure statistically reliable results.
To insure that these two critical questions are answered, once the customer profiling (based upon transaction behavior and household characteristic) is done, smart marketers should divide the file by segments. Only those segments that most match the product and the expected behavior should be used for testing.
Most marketers want to run to test the following:
1. Product, The List, The Offer
2. The size of the test cell(s)
However, there is a third issue to test that is equally important.
3. Was the correct target audience really selected before the list was used or purchased?
Due to conflicting objectives (cost vs. size), the list is sometimes not examined adequately to determine if it really is the specific target audience. When conducting a test, marketers want to use test samples that are as large as necessary to ensure scientific accuracy but as small as possible to keep the cost of the test to a minimum.
The normal marketing behavior is to select a list that provides a close approximation of the "general audience." However, it is vital to move away from the mass mailing and move closer to the targeted mailings which are relevant to the audiences or segments which most closely approximate the profile of the product.
Take this model for example:
Selecting the best audience takes the random selection and guesswork out of choosing which group will respond the best, because it matches the customer demographic, psychographic and behavior to the product or service being sold.
The Product:
Solve the business problem first. Be sure that your business objectives match the potential of the product to deliver adequate profit to support a reasonable roll-out, assuming that you have a successful test. A series of ROI scenarios will quickly determine where the test is warranted.
Even some very large companies have failed in this planning. One example is the breakfast food marketing campaign of a very large food company. The campaign offered a free bonus product when consumers purchased one product. However, marketing forgot that the average consumption of the breakfast cereal was 3.5 months. No testing was done and the campaign was offered to the entire database. The marketing manager is still looking for a job.
The Lists:
Most professionals agree that the list is the most important element in any promotional marketing activity. Since all tests are on a tight budget, how can marketers determine the ideal minimum size to satisfy the test objectives?
1. Select a number from your own database that is projectable if the test proves successful.
2. Test a sufficient number of lists. If you test only one list and the test fails, you won't know if it was due to the list or other factors.
3. If you are purchasing outside lists use a genuine, legitimately recognized broker or a company that is known for its ability to deliver target specific lists.
4. Send all the test mailings out standard mail first class, pre-sort in order to measure and recover and tabulate the deliverability of each list.
The Offer:
The third most important element in any test is the offer. How much should you offer in order to get prospects or consumers to respond? Remember the breakfast food company...Sometimes it take more than you expect to move the customer to action. Therefore, it's important to test various levels of the offer.
1. Analyze whether the offer is financially expandable to a larger audience. Trading margin dollars may or may not be in order to generate a response. Go back to solving the business plan.
2. Consider giving a higher level offer to the "best" customers or the highest prospects based upon their repeat purchase history. This is going back to who is the real target audience. Which target group is the one that offers the best value over a period of time (Lifetime Value).
3. Skip the evaluation of the blue envelope versus the pink envelope, or the clever saying versus the straight proposition on the outer envelope (OE). If you've done a good job of profiling your target audience you already know what turns them on.
Factors to consider when profile modeling is to be used:
1. Predictable Behavior: For a behavior to be predictable the database must include behavior history with the company.
2. The database must contain an adequate number of those customers. There should be at least 500 to 1,000 customers who have demonstrated the behavior you are trying to model for the list purchase.
As an example, let's assume that we are going to send a promotion to our customer base or to a group of prospects who resemble the profiles that have been determined. We could be using direct mail, fax, e-mail or outbound telemarketing. We have several approaches that we want to test. We will set up a group of customers, called "cells," that we will use to test each approach.
For each promoted cell, we will determine the response rate (RR). This rate, of course, is calculated by dividing the number of responders (buyers) by the number promoted (mailed or telephoned). Let's assume that we directed our promotion at 30,000 households and 600 of them accepted the offer. The response rate is: RR = (# respondents) / (# promoted) or 600 / 30,000 = 2.0 percent.
Is this an acceptable response rate? That is determined by whether we made or lost money. What is our return on investment (ROI)? That is calculated by dividing the net revenue by the cost of promotion. The net revenue is the gross revenue from sales, less the cost of the product, the cost of fulfillment, the returns and processing fees.
By calculating the total number of responders, the gross revenue generated from the sale less the variable and marketing costs will determine whether the program is a success. The following illustration demonstrates what happens to a proposed or actual effort.
Using the formula developed above, we can calculate -- in advance -- what level of response we would need from any promotion to just break even. Expressed as a formula, BE is when (net revenue) = (cost of promotion).
This formula tells us the break-even response rate for any combination of outgoing per-piece costs divided by any combination of average net revenue from successful sales. For example, if the per-piece cost of promotion is $5.99 and the net revenue from the average sale is $248.69, then the break-even rate is 1.59 percent.
How Many in Each Cell?
This returns us to the original question. How many consumers must we promote in each cell to be sure that the results we get accurately predict how other similar customers will react to a similar offer? A simple formula answers this question: Minimum cell size = 5.0 / break even. In other words, if the break-even rate is 1.59 percent (as above) then the formula is: 5.0 / .0159 = 314 customers who are expected to respond.
The object of testing is to get a statistically projectable and reliable number. We believe that by approaching the testing in segments that approximate the size of the database after segmentation delivers solid results. Each mailing offer and segment add to 10,000 and the response should be equal to the minimum break even of 159.
We've looked only at what it takes to break even. The next approach is to measure the significance between the break even and the real return. The difference is then assigned an index to describe how well each cell performed. For example, in "Offer One List A" the actual return was 120 and the difference is 40. Now compare to the other cells such as "Offer Two from List B" where the return was 100 or a difference of 20 less than "Offer One from List A." Let's assume that the average response was 30 above the break even. If 30 is the average, then "Offer One from List A" was indexed at 40/30 = 133 and "Offer Two from List B" was indexed at 67.
At first glance it appears that "Offer One from List A" was twice as effective. Now bring in the margin difference and compare the margin return. Margin on "Offer Two from List B" was 10 higher than that of "Offer One on List A." Which one do you think the person in charge is likely to select for the roll out?
There are many types of testing scenarios that have been tried and found successful. This is one. The statisticians out there may look at the database or mailing as homogeneous and would therefore prefer a different type of statistical approach.
I know many readers have experience with tests and controls, return on investment, break-even response rates and minimum test sizes. Their experience and logical framework may differ from the approach suggested here. If you have a different -- and perhaps better -- way of dealing with these concepts, I invite you to contact me so I can share your ideas in a future article.
Thanks to my friend Arthur Hughes for providing some framework for this article.
DBMarkets@aol.com
http://www.msdbm.com
Copyright © 2005-2006 Powered by Custom PHP Programming