The results are in. Last month, we asked you to write the most effective copy for a Consumer Reports email in a way that could also test which value factors were most appealing to Consumer Reports donors.
We’ll get to the results, and the big winner of the MarketingSherpa Summit package, in just a moment. But first, a few lessons for you and a little more background.
Why This Contest?
Every year, we run a test like this with the MarketingExperiments blog audience and the audience of another marketing blog—in this case, Convince & Convert.
We find marketers at a nonprofit for two reasons: we want a real audience to test with, and we want to use our collective ability as a community of marketers to create effective messaging for a greater good. We work with the nonprofit for a few months, diving into the data, getting an understanding of previous tests, and coming up with hypotheses.
Then we go out to you, tell you what we’ve learned about the nonprofit, and give you a chance to win a package for MarketingSherpa Summit (produced by MarketingExperiments sister publishing brand). This year, the prize was a ticket to MarketingSherpa Summit 2016 and a stay at the Bellagio. We award the prize based not on what we imagine will be most effective, but rather on the actual results with real customers. We then use what we learn from this test to run a followup test, crafted by the MarketingSherpa Summit audience. This “live test” is launched and results are reported during Summit.
The goal of this public experiment is to create a tangible example of customer-first marketing to help you improve your own marketing efforts while putting the customer first.
How We Designed the Consumer Reports Value Proposition Test
Test Duration: 01/21/16–01/26/2016
Objective: Increase clickthrough to donation landing page
Problem Statement: The generic value communicated in the renew emails does not match the motivation of why a past donor would want to contribute to Consumer Reports.
Hypothesis: By highlighting specific value claims that communicate the appeal and exclusivity of donating to Consumer Reports, we will increase the overall clarity of value of donating while better matching customer motivations, leading to higher clickthrough to the donation landing page.
Primary Research Question: Which value claim best matches past donors’ motivation to donate again?
Primary Metric: Clickthrough to Donation Landing Page
“Testing four panels [treatments] plus a control is something we haven’t done, as we tend to do one to two test panels at a time. Investing in a test like this, where we took the time to come up with four different themes and testing them head-to-head in one test, is very valuable to have and has inspired me to challenge ourselves to consider doing more robust testing in one round, rather than straight A/B testing in multiple rounds, to get learnings sooner,” said Dawn Nelson, Director, Fundraising, Consumer Reports.
The Copy Treatments
Control: Mixed value proposition
Treatment #1: Honest and unbiased reporting
Treatment #2: Personal impact
Treatment #3: Quality of research
Treatment #4: Consumer empowerment
Results
The control outperformed all of the treatments. The control had the largest impact when measured against treatment 4 (29.2% relative decrease at 99% level of confidence), while it did not validate when compared to treatment 3 (6.7% relative decrease at 85% LoC).
What You Can Learn from This Experiment
There are two levels of lessons you can gain from this experiment. At the base level, there is the analysis you must conduct with any test: why did one treatment perform better than the other? Or in this case, why did the control win?
“This testing reaffirmed that our core donor still responds best to messages around product testing, and that is their main connection with Consumer Reports. Asking for a donation around this theme still works better than messaging around empowerment or our impact work in particular (to the broad audience),” Dawn said.
“The big takeaway for me on our customer (and something we have suspected and need to test further) is we definitely can talk in a broad sense to our donors and see good results. But if we can do segmentation and talk to those segments in very specific language, we may see better conversions,” said Bruce Duesterhoeft, Program Manager, Online Fundraising, Consumer Reports.
This public experiment was essentially a mini Research Partnership between the team at Consumer Reports and MECLABS Institute (parent research organization to both MarketingExperiments and MarketingSherpa). According to an analysis from the MECLABS team (major props to Melissa Mike, Jesse Kraker, Derek Snow, Jonathon Yates, and Jordan Baker for their work on this experiment) based on the patented MECLABS conversion heuristic, here are a few key lessons:
Value
- The value proposition for the control focused on multiple value statements throughout the email, which could have appealed to the motivations of a wider user base over the treatments which focused exclusively around one value statement.
- The control and treatment 3 focused on the quality of research, increased clarity about Consumer Reports testing by using specificity surrounding the testing process, and how Consumer Reports would use donations. The other treatments did not provide specific value points which could have hurt the clarity of the value proposition.
- The value proposition of treatment 4 (consumer empowerment) did not provide as many specific evidentials to support the claim of consumer empowerment which reduced the clarity and appeal of the email, causing it to perform the worst of all treatments.
- The opening statement of the control may have provided more relevance to customers and provided continuity from previous customer touchpoints.
Motivation
- Across the control and all treatments, the most clicked links were the red “Donate” button CTAs on the top and bottom of the page. However, the next most clicked link was the membership benefits link in the footer of the email. This shows that length-based friction may not be a deterrent for this group base and that many are motivated to learn about benefits membership provides.
- Those motivated by the value of the testing and personal/impact are the most likely to donate more money; however, there are less Consumer Reports donors motivated by altruistic outcomes than there are by the product testing process.
Friction
Both the control and treatment 3 used very similar layouts that mix bullet points, paragraphs, and text links. This layout reduced difficulty-based friction associated with longer emails, providing clarity for customers trying to absorb the information being presented and understand the overall ask of the email.
“Learning about the value proposition really opened my eyes to looking at our work (and other marketing, too) with a different eye,” Bruce said. “We definitely will be reviewing all of our emails and landing pages using the probability of conversion equation [heuristic] moving forward.”
[contextly_auto_sidebar]
Challenge the Model
The previous lessons apply specifically to this test, but there is a bigger lesson to be gained from this public experiment.
When I first approached Dawn and Bruce about partnering on this public experiment, they were already successful marketers and very experienced testers. As Bruce said in the blog post that launched this experiment, “Since 2013 we have grown from $800,000 a year to looking at over $2.7M this year.”
But they didn’t settle. When approached with a very out-of-the-box idea, they jumped in wholeheartedly—and I can attest, throughout this process, made many difficult decisions that challenged their current understanding of their customers.
That’s no easy task. We work so hard on our brands that we assume we know our customers. Perhaps we’ve already tested heavily with our customers. To publicly go against the wisdom we think we have already gained about our customers in a campaign (particularly one that’s expected to generate revenue) goes against every bone in our marketing bodies.
Never be complacent. The times change. People change. Our competitive landscape changes. Always challenge your current model. These difficult challenges are where you’ll find the next opportunities.
You might sometimes get a loss, but you will always get a greater understanding of the customer (if you experiment correctly). And in the long run, that is how you achieve sustainable business results.
“Challenge what you think you know by doing very specific testing around messaging (first) and then presentation (second),” Dawn advised. “Test the boundaries to ensure there are very distinct differences in your messaging to allow you to really discern what is resonating and what definitely is not. You can learn as much from the ‘losers’ as you can from the ‘winners.’”
“Marketers need to leave their ego at the door and look at their messaging with a different eye. Our instincts are not always the consumers’,” Bruce suggested.
And the Contest Winner Is…
While none of the treatments outperformed the control, we were also holding a copywriting contest, as I mentioned previously. The winner of the contest is the writer of the best performing treatment, even if it didn’t beat the control. After all, these intrepid copywriters were creating a treatment against a control created by experienced marketers with a deep inside knowledge of their brand.
When comparing the treatments against each other, Treatment 3 (quality of research) outperformed all the other treatments with the largest increase in clickthrough compared to Treatment 4 (24.1% increase at 99% LoC). So congratulations to the writer of treatment 3: Ahnna Pildysh, product marketing freelancer, Malaspina Labs. Ahnna won a ticket to MarketingSherpa Summit 2016 and a stay at the Bellagio Resort.
“Since Treatment 3 themed around ‘quality of research’ was second to the control in terms of clickthroughs and donations, I learned that this theme resonates and is one we haven’t emphasized as much in past communications and perhaps could emphasize more in our future messages, along with product testing,” Dawn said.
When asked about her approach, Ahna said, “I chose to use two value focus options which were based on the market research doc that was provided. I focused on the value trigger points of the donor audience and wanted to deliver simple messaging that highlighted how Consumer Reports delivered on those expectations. The market research doc was really helpful in understanding what the audience cared about and helped me narrow down my focus and keep the value prop on target.”
“This contest piqued my interest simply because it gave me an opportunity to test my ability to craft a value prop. I’m a big believer that everything should be tested to gain useful learnings and continuously iterate,” Ahna said.
Congratulations again to Ahna, and thanks to all of the marketers and copywriters who took the time to share their wisdom and provide insights Consumer Reports can use to better serve donors and consumers.
Get more content like this, plus the very BEST marketing education, totally free. Get our Definitive email newsletter.
