Here’s an interesting (though oddly titled) post by Jon Richman: Lies, Damn Lies and Pharma Social Media Statistics.
It is interesting because it beautifully un-packs misreporting on a topic of great interest to e-patients. It is oddly titled because while the pharmaceutical industry is part of his beat, the errors in reporting on surveys he describes are common across the board.
Caveat emptor – and thanks, Jon, for so clearly articulating the caveat.
(Editor’s note: Jon is @JonMRich on Twitter.)
Please consider supporting the Society by joining us today! Thank you.
I’ve been traveling so much, this whole subject didn’t get my attention, even when a half dozen friends tweeted about it. And thanks to you, Jessie, for drawing it to our attention.
Folks, Jon Richman has written an exemplary indictment of what I’m willing to say is a very poor piece of research by the National Research Corporation. It seems to be purely promotional, with little rigor in the method or the reporting. Worse, a significant consequence is that many blogs and media outlets picked it up and have blindly reported the headlines, without checking into the method.
Which apparently stinks, as Jessie suggests (more politely) above. So the net result is a decrease in our collective knowledge about reality.
NOT good.
NRC is a marketing firm. I worked most of my life in marketing, so I understand promotion and I understand being responsible. I don’t know any of the people so I hesitate to speak too strongly but this is NOT a well designed, well reported objective survey. Pew, for instance, displays their questionnaire, along with the raw data, and explains exactly how they calculated their results. This is important, because it lets others check for mistakes.
In contrast, this study’s methodology page doesn’t at all disclose what they asked, so there’s no way at all for an informed, empowered consumer (or policy analyst) to assess its relevance to their situation. That’s disempowering and should be avoided by anyone who cares whether a reported factoid can be relied on.
(Empowered people take responsibility for assessing what they read.)
Then, the methodology page says the data was crunched “according to an innovative and thorough tabulation specification plan.” Speaking as a marketing guy myself, I can say THAT is marketing BS if I ever saw it. C’mon, guys, what did you do?? Or are you just trying to sound impressive to potential customers, who might not be bright enough to ask the same question? (Sorry if I’m being harsh, but what could possibly be innovative about a tabulation plan? And if your findings can’t be scrutinized, will anyone sensible trust them?)
Of course, the answer to that is that people trust empty headlines all the time. But we’re here to warn people away from that: whether it’s a medical journal article or a Facebook survey, people should ALWAYS dig beneath the headline.
The punch line in this case is that although they did send Jon the results spreadsheet, it revealed a gross Excel error: a column is formatted wrong, so average age 40.8 is displayed as 40.8% of all users! See the screen capture in Jon’s post, at Issue 6:
“Average Age,” displayed as a percent?? What statistician published THAT? What manager or editor scrutinized it before publication?
And it was compounded by the many people, even including our friends at CNN, who trusted too quickly and didn’t ask and verify as Jon did. The NRC press release (at least as of today) says 41 is the average age, but CNN’s blog says “41% said they use social media as a source of health care information.”
Here’s hoping they all post corrections soon, with a hat tip to Jon.
=====
Here’s the counter-example. Jon Richman did what I always do – turn to an objective researcher –
“So, what’s the actual number? How many people actually use social media for healthcare information? I looked to Pew Research and Susannah Fox for this since I’ve been through nearly every single one of their studies and dissected each and consistently find the highest standards of research practices. I asked Susannah about her views on the National Research study and she had this to say:
As Jessie said: e-Patient, Beware. Ask questions, and think for thyself.
Outstanding comment, Dave. I should have had you as co-writer. You added some nice additional perspective here, which I didn’t include simply for brevity (such as it is with any of my posts).
> I should have had you as co-writer.
Naa, it’s important for us all to forge ahead. I have such a backlog of things I want to write about, it’s a GIFT when you and Jessie do something like this – it’s easier to comment on someone else’s work than to do the work in the first place. :-)
Thanks for the write up of my post, Jessie. You’re right about the title. About 20 seconds after I posted it, I thought better of it and wished I left out the “pharma” part.
Well you CAN edit the post title, y’know. :) Especially if nobody’s seen it yet. I usually do have a second thought, shortly after hitting the send button.