Social Media Research

Remembering the Dangers of Social Media Research

Jay Baer Blog Post

This one is true. I just know it.

Numbers don’t lie. But sometimes we lie to ourselves about what numbers represent.

Last week, I wrote a post about new research from ExactTarget that studied Americans’ digital channel preferences. It was a lengthy report, but what stood out for me was the finding that only 5% of us prefer promotional offers from brands to show up in social media – even brands we have given permission to contact us. I accompanied the post with the somewhat inflammatory headline – New Research: Americans Hate Social Promotions. This is not factually correct, as the survey did not ask about love vs. hate, but rather about “preference.” I should have been more careful about that.

What I found even more interesting than the research was the negative reaction to it by readers at Convince & Convert, on Twitter, on Google + and elsewhere. While it had several somewhat strident tentacles, the gist of the complaint was “this research cannot be true, because I clearly remember a study saying more than half of all people WANT promotions and offers in social media.” Not a single aggrieved reader cited the research numerically or with a link, but several were convinced that consumers’ desire for special offers and coupons in social media was as insatiable as Ozzie Guillen’s lust to say stupid things.

They remembered the research in general. It stuck with them. But they couldn’t remember the company who published it, what the exact numbers were, or how they were divined. I remembered it too, and looked it up. It was the Lithium and CMO Council study on the Social Brand Experience from December, 2011.

Upon review, however, these reports are not contradictory. If you ask two different questions, you’re likely to get two different answers. Here’s how:

ExactTarget 2012 Channel Preference Survey

Survey Sample: 1,481 Americans, using MarketTools database, randomly sampled and weighted to be representative by age and gender

Response Methodology: Written survey, with permission granted by parents for participation for minors

Question Asked:

“What is your preferred channel for promotional messages from companies whom you have granted permission to send you ongoing information?”

Lithium/CMO Council Social Brand Experience Survey

Survey Sample: 1,300 consumers. Unclear whether they were all Americans, or whether they were existing social media users. Lithium published age and gender demographics, which were not weighted to match national patterns. Uncertain if this was random sample, or self-selected participation

Response Methodology: Online survey

Question Asked:

When I like a brand on Facebook, I expect:

Multiple answers were permissible, and 67% of respondents selected “To be eligible for exclusive offers”.

Well yeah. I’m surprised it wasn’t higher, actually. If I Like a page, I do suspect that they will give me a special offer at some point. That doesn’t mean I prefer those offers to be in social media versus other places.

The Danger of Data

There are at least 4 ways these research reports are different, and consequently why the findings of each are not in conflict.

1. One survey asked about preference. The other asked about expectations

2. One survey allowed for one answer. The other allowed for multiple answers.

3. One survey asked about Facebook in comparison to other channels. The other asked about Facebook “like” outcomes

4. One survey asked about how consumers behave. The other asked about how consumers anticipate brands will behave.

5. There are also significant differences in sample and response methodology.

Given that in the same Lithium results set, 50% of respondents said that they like a brand on Facebook to find service and support, and 41% like a brand on Facebook to share their ideas for new products, do you believe that companies are massively under-executing in the areas of Facebook CRM and crowd-sourcing? Both of those are interesting findings, and just as valid as the “67% of fans want special offers” data point. But the latter was the headline on dozens of blog posts, and maybe we vaguely recall the headline and the Mashable article, and maybe we start to make business decisions based on that piece of information. And then maybe we reject out of hand another data point that we assume is incorrect because it contradicts what we’ve convinced ourselves to be true.

This isn’t about which research is better, or “true”. They are both good studies using different approaches, and there’s no such thing as “true” research, just accurate interpretations of it.

Business Relationship

Facebook Comments


  1. says

    I like to think of surveys as another spin on the “liars, damn liars, and statisticians” quote. Without knowing all the details of the questions asked, the sample group asked, and all the details behind it you only get the interpretation the group providing the survey wants you to see. However, I’m a natural skeptic but that’s only because I know how/what/why companies use these to alter perceptions.

  2. says

    The way you conduct research you can make a study find anything. It is always important to read past the headline and see how the research was actually done! 

  3. margieclayman says

    60% of the time your posts are awesome every time :)
    Social Media research is definitely tricky – one can easily pull stats favorable to his or her point of view and boom! You have expertise or a proven point or huge news. We always have to do our due diligence, non?

  4. avi.kaye says

    The problem is, that there are so many ways to game surveys, it kind of makes you wonder what’s the point any more. For example, I might point you in the direction of this survey conducted by a company you might recognise, which states that 58% of the people expect to receive discounts or promotions via Facebook (link at the bottom).
    Yes, I know it says ‘expect’ and not prefer, but for those of you who can’t be bothered to follow the link, the survey was conducted by ExactTarget 6 months ago. So I might expect the way questions are asked to be a bit more similar, so I can actually compare the two surveys done by the same company, giving me contradictory results, but maybe the surveys here were intended for different reasons. And yes, when a survey tells me that users ‘expect coupons on Facebook’, it contradicts a survey saying that ‘users prefer to get coupons via email’, even though the verb and action aren’t the same. Sorry, but the rest is semantics. 
    By the way, I will re-iterate my comment on your previous post – it’s not that I don’t agree with the fact that you need to provide your fans with good content (I think that by now that’s a given). It’s just that surveys should be there to help me understand what my users would prefer to get, and they don’t.

    • says

       @avi.kaye Actually, that’s not surprising at all. That previous ExactTarget research – “The Meaning of Like” (which I analyzed on this blog) asks the question almost exactly the same as the Lithium study. 
      Totally different study from ExactTarget, using a different methodology, and a completely different question. I really do not see your point. Let me put this another way.
      Imagine I asked 100 people which they would prefer for dinner: hamburgers, pizza, or ice cream, and 5% of them said ice cream.
      Now I’ll ask those same 100 people what would they expect a restaurant to serve, and I gave them a multiple choice list of options that included ice cream. 67% of them checked the ice cream box. 
      Is that data in conflict? No. And that is EXACTLY the situation we have here with the recent ExactTarget research vs. the Lithium and prior ExactTarget research. 
      You say the “rest is semantics” but it’s much, much more than that. It’s ALL about semantics. 
      How are these surveys “gamed’? The gaming that’s being done is by the readers of the surveys trying to interpret the data to fit their own narrative. And that’s where we get into trouble, why it’s a dangerous game, and why I wrote this post. 

      • avi.kaye says

         @JayBaer  It’s not that I don’t get the difference between ‘prefer’ and ‘expect’, although your example did make the difference more clear. It’s that the surveys are gamed by the people asking the question – and by the people writing posts with eye-catching headlines, by the way – in a specific way as to get the answer they were looking for at the time. 
        For example, I can assume, reading your original post, that people prefer getting their coupons (we’ll call it coupons, but of course I’m referring to offerings as a whole) via email. Assume? Hell, that’s what the data SAYS. I can’t interpret that data any differently! Away to my email sending machine!
        But that, of course, is wrong. If you had given the same question (what do you prefer, as opposed to what do you expect) and offered MULTIPLE CHOICE (ie. making sure the playing field is level), then, if only 5% ticked off ice cream, then the survey might be more valid.
        But that didn’t happen. The questions were changed, the format was changed, and a different answer was (surprise, surprise) given. The survey is gamed. The data is skewed. 
        We get into trouble not from interpreting the data (or rather, not just from), but also from the fact that the data is presented in a very problematic form.
        All things being said, I’m not saying that this survey is bad, or that ExactTarget is out there gaming surveys for their private gain, far from it. I’m just saying that after reading your posts, I come away with a strong feeling that no one knows WHAT people expect from businesses in social media, be it email, Facebook, LinkedIn or whatever. And that these surveys, the more you dig into them, with each one saying something different, seem only serve to muddy the waters. 

        • says

           @avi.kaye Having a different survey with different questions isn’t “gaming” a survey, and the data is not skewed. The data from one survey has NOTHING to do with the data from another survey. That’s my point. 

  5. Ted Sindzinski says

    Ask people to check what they like from a list without any specificity and what you get is an insight into no one in particular. While I’ll admit to having quoted many social stats to make a point, as they become more plentiful, more absolute, they worry me. By definition engagement is not about the “big picture”, it’s about your picture.
    What I’d like to see is some research on single brands. What motivated people to like them in particular? What are they expecting to see, if anything? And how does it correlate back to the brand’s value proposition elsewhere…

    • says

       @Ted Sindzinski Very good call Ted. I’d love to see that research too. It would also make the “A Facebook Fan is Worth $xxx” nonsense more palatable, because you could justify what a fan was worth to THAT brand. Once you add a second brand to the calculation, you invalidate the research. 

    • says

       @Ted Sindzinski Very good call Ted. I’d love to see that research too. It would also make the “A Facebook Fan is Worth $xxx” nonsense more palatable, because you could justify what a fan was worth to THAT brand. Once you add a second brand to the calculation, you invalidate the research. 

  6. coreycreed says

    Very well explained, @JayBaer .  Thanks for this.  
    I think the biggest problem our industry faces with statistics in general is that websites like Mashable and RRW simply throw out strong headlines which get retweeted and shared like crazy.  That’s why people tend to remember the general idea (especially the ones they like) but not the true source or method of arriving at these statistics.
    In other words, sometimes we need to use our brains and not just glance at blog headlines to make credible decisions.  If all of our social media training is from reading blogs, we’re all in trouble, IMHO

    • says

       @coreycreed Exactly. I’m not blaming Mashable or RWW, or anyone else, but just cautioning that we need to realize that they are in the eyeballs business, not the nuanced analysis and accurate research business. 

    • says

       @coreycreed Exactly. I’m not blaming Mashable or RWW, or anyone else, but just cautioning that we need to realize that they are in the eyeballs business, not the nuanced analysis and accurate research business. 

Leave a Reply

Your email address will not be published. Required fields are marked *