{"id":13374,"date":"2012-10-05T15:34:38","date_gmt":"2012-10-05T19:34:38","guid":{"rendered":"http:\/\/pmedicine.org\/epatients\/?p=13374"},"modified":"2012-10-05T15:34:38","modified_gmt":"2012-10-05T19:34:38","slug":"fact-checking-at-medicine-x","status":"publish","type":"post","link":"https:\/\/participatorymedicine.org\/epatients\/2012\/10\/fact-checking-at-medicine-x.html","title":{"rendered":"Fact checking at Medicine X"},"content":{"rendered":"

(A cross-post from susannahfox.com)<\/em><\/p>\n

I had the great honor of being part of the first Medicine X<\/a> conference at Stanford University last weekend. I presented a sneak preview of new survey results collected by the Pew Internet Project and the California HealthCare Foundation. Overall, the conference was magical, as I wrote in a previous post<\/a>.<\/p>\n

In this post I would like to explain why I interrupted the HealthTap presentation.<\/p>\n

\"Fred<\/a>

With thanks to Fred Trotter for adapting the original cartoon: http:\/\/xkcd.com\/285\/<\/p><\/div>\n


\nEarlier this year, a reporter contacted me to ask about some Pew Internet data that HealthTap was using on the front page of its site and in slide decks. Pew Internet welcomes accurate citation with attribution. That is a common use of our research, whether it is a for-profit, a non-profit, or a government agency making a case for its work.<\/p>\n

Unfortunately, HealthTap\u2019s interpretation of a data point was not accurate (more about why in a minute). I wrote to Ron Gutman, founder and CEO of HealthTap, and he responded pretty aggressively that he stood by his own interpretation of the data. I was taken aback. I have never had someone tell me I wrongly interpreted my own data and I wasn\u2019t sure what to do next.<\/p>\n

I corrected the reporter’s impression of the data and decided that was the approach I would take if I were asked about it in the future \u2013 fact checking on demand.<\/p>\n

Here are the facts about our findings: Pew Internet has historically asked a pair of questions meant to measure the broad impact of the internet on health in the U.S. We have asked the questions of all adults, not just internet users. That respondent audience is important, so keep it in mind:<\/p>\n

Have you or has anyone you know been HELPED by following medical advice or health information found on the internet? [If they answered yes, the respondent was asked: Would you say the information provided major help, moderate help or minor help?]<\/p>\n

Have you or has anyone you know been HARMED by following medical advice or health information found on the internet? [Again, if yes, the respondent was asked a follow-up question: Would you say the information caused serious harm, moderate harm or minor harm?]<\/p>\n

Some respondents get the \u201chelp\u201d question first in the sequencing of the questions, others get the \u201charm\u201d question first, to minimize bias.<\/p>\n

People who answer “no” to the \u201chelp\u201d question represent a population who, at that time, do not know anyone who was helped by information found online. They may have done a health search and found relevant information, but may not feel strongly enough about it to say \u201cyes.\u201d Or they may have looked online for medical advice and not found anything useful. We don\u2019t know. That’s the beauty and the curse of a poll. All we know is the number of people who say the internet has been a help. It is not a question about the internet\u2019s \u201chelpfulness.\u201d<\/p>\n

The same goes for the \u201charm\u201d question. The respondent is asked to recall if they or anyone they know was harmed, not to rate the \u201charmfulness\u201d of online information.<\/p>\n

Our findings have fluctuated over the years, so let\u2019s just look at the data cited by HealthTap, from our report, Chronic Disease and the Internet<\/a>, which is based on a 2008 national phone survey<\/a>:<\/p>\n

\"Pew<\/a><\/p>\n

Looking back at this infographic, I have to admit I don\u2019t like it. I can see how HealthTap could interpret it the way they did, emphasizing the \u201cno help\u201d finding.<\/p>\n

However, they take it a step farther and imply that people were rating the \u201chelpfulness\u201d of online health information:<\/p>\n

\"HealthTap:<\/a><\/p>\n

HealthTap\u2019s interpretation distorts the meaning of the question. The people who answered \u201cno\u201d were not saying that the internet was unhelpful. They were in all likelihood saying they do not happen to know anyone who has been helped.<\/p>\n

Also, one-quarter of adults in the U.S. didn\u2019t even have internet access in 2008, when this survey was fielded, and 20% of internet users had never looked online for health information. There is simply no way to slice the data to fit HealthTap\u2019s conclusion.<\/p>\n

Again, looking back, I don\u2019t like our question. It is too vague. Our infographic does not display the data well. I do not fault anyone for making a mistake and misinterpreting the finding. What is troubling is HealthTap\u2019s refusal to consider my explanation.<\/p>\n

Fast forward to Stanford Medicine X.<\/p>\n

I have seen Ron talk about HealthTap on multiple occasions and I know the pattern of their pitch pretty well by now. On Saturday, when Geoff Rutledge of HealthTap took the stage, I was interested to learn more about their mobile platform and growing network of doctors. But then the familiar patter began. He was warming up to the same pitch I\u2019d advised Ron to stop making. And there on the screen was the same incorrect table being described as \u201ca recent study from Pew Internet.\u201d I couldn\u2019t let it pass, so I raised my hand and said in a loud voice, \u201cThat\u2019s my study and that\u2019s actually not how I would interpret the data.\u201d Geoff paused, said we could discuss it later, and went on to his next point.<\/p>\n

I immediately felt bad about interrupting Geoff. It is not my style to be disruptive and, frankly, rude. I took to Twitter to explain myself in the back channel:<\/p>\n