Social Media Case Studies

Our Dangerous Addiction to Social Media Case Studies

The two most important words in social media should be “so what?”

As Tom Webster eloquently put it in his recent Blogworld keynote, when research wears the cloak of content marketing, it’s a recipe for the incurious to pull a fast one on the masses, disguising pointless data as gospel.

Tom was referring to data-dredging and the spate of statistical “best practices” that bubble up in social media like something in Jed Clampett’s back yard. But it applies equally to the basket of case studies that are left for us each day by the sites that chronicle social’s advance. It’s like Moses in the bullrushes but with “math” instead of blankets.

The Recalcitrant Social Media Cheerleader


Point Less

Too often, we try to wield these case studies as a crude evidentiary weapon, trying to persuade ourselves or our superiors that our belief system around social media, social business, social CRM and all that jazz is in fact correct. But in almost every case, you’re bringing a knife to a gunfight. Or maybe even a limp noodle.

Case studies should be used for ideation, not ratification. Even in the best possible scenario, where the case study in question is extraordinarily applicable to your business goals, social media situation, KPIs, budget, timeline, customer personas and more (which is a rare alignment indeed), you are placing significant influential value on ONE outcome. That’s anecdote-driven decision making at its worst.

If you only need to hear one story that you find persuasive, I can convince you of ANYTHING if I get to select the parable.

Beyond the fact that case studies are often strategically irrelevant because the company profiled is in a different industry, with different goals, competitors, and customer expectations (among other variances), perhaps the biggest problem with most social media success stories is that the measures of that success are largely without real merit.

Social Media Case Studies – The Crack of the Stat

My favorite example last week was the glowing report on ClickZ of Louisville Slugger‘s scavenger hunt promotion. Evidently, the erstwhile manufacturer of wood and aluminum bats launched (with agency help) a one-day promotion on October 29 in St. Louis. They hid 45 commemorative World Series bats around STL and posted clues on their Facebook and Twitter accounts as to their whereabouts.

I’m no baseball manufacturing expert (but I do adore the Louisville Slugger Museum in downtown L-ville), but I’m guessing they ultimately need to sell bats to kids and weekend warriors to maintain/expand their business. The MLB buys what they buy, and the Twitter acumen of the company has a zero percent impact on whether Derek Jeter swings their wood.

Strategically then, the question becomes “does hiding and giving away 45 bats and talking about it in social media ultimately sell more bats to amateurs (either current or new customers)?”

And the simple answer is that we do not know. This case study appeared 10 days after the conclusion of the promotion. Is 240 hours enough time to determine the true impact of this promotion? Of course not. November is hardly peak season for bat sales anyway, and any bottom-line impact of this campaign would need a while to germinate and flower.

But that doesn’t stop the agency (presumably), the company, and ClickZ from proclaiming the unmitigated success of the effort. On what basis? Evidently, Louisville Slugger saw a 834% increase in the “talking about this” metric on Facebook, which purports to measure total number of Facebook users referencing the brand in any capacity whatsoever.

Well, if you go to 45 locations around town in one day and give stuff away, a goodly portion of the people angling to be the recipient of free goods will be updating status, location, photos and so forth. It’s what we do now, and our proclivity to shout from the status rooftops about the awesome thing we’re engaged in at this very moment is exacerbated by three factors: deviation from the hum drum norm; a crowd; and free stuff. Wisely, Louisville Slugger designed a program that hit for the cycle, and St. Louis residents choosing to spend a portion of their Saturday chasing after free baseball bats responded on Facebook en mass.

But does a 834% increase in people “talking about you” on Facebook achieve business goals like sales, retention, average order size, advocacy, etc? Again, in this instance we do not know yet. Before this initiative (and I give them credit for divulging real numbers) Louisville Slugger had 755 people talking about them for the week. Afterwards, it was 7,049.

The brand spent considerable time and effort to drive a short-term spike in conversations about it on Facebook. So what? You might argue that because they also saw 100%+ week over week increases in Facebook fans and Twitter followers, that this campaign was a smash. Again, so what? They have 21,000 Facebook fans, and their best posts (announcing a second giveaway) show an engagement rate of .03.  They have 3,200 Twitter followers, and the account essentially parrots their Facebook status updates.

I’m not suggesting this is a flawed campaign or wholly without merit. We simply do not know whether it has strategic or financial merit, and I’d love to see a case study that tracks sales and customer retention in the St. Louis market on a 60-day trailing basis. Now THAT would be a case study.

I am not irked by the idea, or the effort behind it. I am irked by the near-instant reporting and self-congratulatory reach around that content repositories, agencies, and corporations are engaged in every time they find some statistic that allegedly proves that their idea was a big success. And I’m irked that we fall for it every time, retweeting and sharing and printing it out for our boss to read on Monday.

We can do better than that, can’t we?

The poll mechanism above is a Quipol. Quipol is a brand-new, supremely easy-to-use polling tool created by my friend Max Yoder. It includes embedded comments (maybe you left one) and a handy thumb up/down function. It’s built for bloggers (easy to embed), and you can also drop Quipol’s on regular Web pages and elsewhere. The comments can also be shared to Facebook or Twitter.

Quipol is totally free, with advanced functions rolling out for small add-on fees. Check it out.

Max is an awesome guy, and an inspiration. He’s 23 and already paid off all his student loans and is bootstrapping Quipol on his own nickel (no VC) by saving 61 percent of everything he makes at his day job (working for Compendium – a former client here at Convince & Convert).

Facebook Comments


  1. pbehnia says

    Great post – I’m waiting for people to break out of the social media echo chamber in which they’re in. This example you raise tells me that we’ve forgotten that the fundamentals of marketing don’t change – it’s that the tools change. I wish we’d be grounded in the basics while leveraging the new and awesome.

    • says

      @pbehnia 1000% true. Tools always change. The rest doesn’t as much. My favorite is people coming up with new ways to calculate ROI, when ROI is a defined formula. No conjecture or modification warranted.

      • webby2001 says

        @JayBaer@pbehnia I’m coining “ROJTSOSM,” which is “Return on justifying time spent on social media.”

  2. be3d says

    Agreed, but let me elaborate. I can tell you spent a lot of time thinking about why the case study above was junk. In the screenwriting world, they say it’s just as important to read bad scripts, that you learn just as much, if not more from the plainly awful.

    Mark Cuban says, “all it takes is one idea.” So you could have a case study that’s 99% anecdotal crap, and still be enriched by reading it by virtue of that gleaming 1%. Still worth they time you spent considering it, even if it only tells you what not to do.

    • says

      @be3d Absolutely right Ian. As I tried to convey – perhaps with limited success – I very much liked their concept. Good idea, and by all accounts executed well. It’s the cloak of math around it that bothers me. If it was a case study heralding the idea, I’m in their corner. But it’s not. It’s a case study heralding the “results”, and it’s too soon to make that call.

  3. webby2001 says

    Thanks for the “invocation,” Jay :) And, in exposing the danger of anecdotal case studies, you’ve chosen a great anecdotal case study. My favorite line: Case studies should be used for ideation, not ratification. Case Studies, per se, are marketing tools for content marketers. They seek to confirm, not to disconfirm, and thus they’ll always hit their target.

    Will check out Quipol, too – looks cool! Though I think you’ve led the witness on today’s question :)

  4. DavidBThomas says

    I completely agree that the social media world spends far too much time quoting stats that are either meaningless, unrelated or yet to be proven. It will continue as long as the people who do social media for a living, whether in companies or agencies or as consultants, are pressured by the people who sign their checks to produce instant ROI. It’s harder to wait until all the results are in when the boss is waiting impatiently, and maybe threatening to move the budget dollars somewhere else.

    • says

      @DavidBThomas The great paradox is that social media is the fastest thing ever, yet results accrue slowly. That’s the trade-off for relevancy, and winning hearts and minds one at a time instead of many at a time via push marketing.

      • philsimon says

        @JayBaer@DavidBThomas No argument here. For every Dave Carroll, there are thousands or millions of us who plug away with little or no **immediate** return. Great post, Jay.

  5. casey_hibbard says

    I write a monthly case study for Social Media Examiner and I can tell you the decision of who to feature is tough. The readers continuously push us for more bottom-line numbers to take back to their companies but those types of stories are very few and far between. And the people I interview say those numbers aren’t that important to their companies – yet. In the absence of that, I go with ideation. Which stories offer new strategies we might add to our toolboxes? I’m taking your point about timing to heart. Most of the pitches that come in are from very recent campaigns where the real measure of success probably won’t come until later. Thanks!

  6. carmenhill says

    Great post, Jay! I confess to squirreling case studies away like acorns for the winter, so I’ll be prepared when people ask whether there are any examples of companies with tangible social media results. (Um, yes.) But as you articulated so well, that’s the wrong the question. What we need are relevant examples of companies in similar industries with similar objectives. Otherwise, the answers are interesting and even inspiring, but don’t really provide data we can apply to our own work.

    • munnerlyn06 says

      @carmenhill What if your industry is the first of it’s kind on social media? I work for a screen printing company and I tell you what, it has been difficult to find tangible social media results of other screen printing companies…

      • says

        @munnerlyn06@carmenhill Good point. We’re still at the point where there’s not a lot of applicable case studies in some categories of business. You can be the first, and you need to have the courage to do so.

        • carmenhill says

          @JayBaer@munnerlyn06 That’s also a really good point. We have had clients ask whether any other companies in their space are blogging or using social media or whatever, and my response is that there’s power in being the leader.

  7. BigBadMN says

    Well said, Jay. I will share this with the next person who drops by my cube trumpeting some “brilliant” social media example with so-called tangible results they saw last week and saying “We HAVE to do a campaign exactly like this starting next Monday”. I agree with what you and others have stated. I most definitely appreciate very creative ideas gathered from others’ recent social media campaigns. But I’m not hanging my hat on their results.

  8. says

    I think your example case study is one of the recurring issues around social media adoption and measurement, it’s not integrated with other business metrics. If this company was in the business to sell “talking about metrics’ then yes, their campaign was a huge success. Until social media is embedded into how a company operates we’ll continue to see case studies like this. A more effective case study might be one that describes the goal of the effort and describes how social media, in conjunction with other tools, impacted a specific set of metrics. I wonder if the agency and company have built in trackers to their other online assets that will enable them to tie future sales to this social media campaign?

    Thanks for sharing.

    • says

      @Collectual I’m going to guess that they do not have that type of tracking ability. Admittedly, doing so would be somewhat difficult and expensive. But it would be much more valuable data.

  9. Chris_Eh_Young says

    Great post. Let’s finally get to the bottom line and stop with the craptastic case studies that are skewed in the agency’s favour.

  10. margieclayman says

    Gosh darn it, Jay. This post really ticks me off. I didn’t think I could ever get mad at you, but here we are.

    I mean, how am I ever supposed to disagree with you when you are so sensible? Seriously, you’re making it really hard for me to call you out, say that you’re spouting nonsense, or any other such thing, and that’s really foiling my plans for world domination.


    I guess all I’m left with is, well, this is a brilliant post and uh, well, thanks for writing it.

    But I’m REALLY mad at you!! :)

    • says

      @margieclayman Ha. Thanks Margie. Means a lot coming from you. You are the epitome of common sense, so if I seems common sensical to YOU, I’ve done my job.

  11. brycehomier says

    Great point. I just read a book called Social Media, and while the book didn’t do a lot for me and my personal brand, it had a lot of good things to say similar to what you have to say. Reading metrics when it comes to Social Media ROI is near impossible.

  12. says

    Agree and disagree.

    You say they are premature in calling this promotion a success because they don’t have information about sales yet. How can they tell if it’s a success if they don’t know if it has sold any bats yet? I agree with that. Those kinds of business figures would make this case study much more valuable. That being said, if the first step of the campaign was to merely get people talking, get more fans on Facebook… then yes, this was a success.

    Also, I guess I could do some digging to find this, but was the case study published by the client? or the agency? If it came from the agency then I think it is totally appropriate. Chances are, they were hired on to conduct a promotion, to increase brand awareness on social media, or something of the sorts. The agency did what they were hired to do. And thus, it was a success for them.


    (And full disclosure… I’m from STL, work at a local marketing firm, but had no idea this promo was going on until now.)

  13. adpartain says

    Good points mentioned in here. I think that you can’t always rely on other peoples results to prove that your social media campaign will work. Almost all campaigns are different and their success depends highly on whether or not the social media is run properly. I just finished Social Media ROI and it was a great book for anybody wanting to break into the social media field and hit the ground running.

Leave a Reply

Your email address will not be published. Required fields are marked *