A new study in the Journal of Medical Internet Research (JMIR) suggests that inaccurate medical information is easily found and regurgitated by students. But the topic the study chose to study — vaccines — has been under increasing scrutiny and controversy, limiting the generalizability of the results.
The study was simply setup — 34 high school students were asked to find websites about vaccines, and then rate their accuracy. Of the 40 sites found, 22 were described by the researchers as “inaccurate,” including well-known sites such as CBS News. The search term “vaccine dangers” returned virtually all inaccurate search results out of the first 20, while the search term “vaccine safety” returned virtually all accurate search results.
I think the only conclusion we can draw from a study of this nature is that controversial medical topics — whether the controversy is warranted and valid or not — brings out a bias in search results as dozens of new sites pop-up to discuss and offer their own viewpoints on the controversy.
Less surprising is the fact that high school students couldn’t differentiate an accurate site from an inaccurate site. There is no single indicator (or even set of reliable, research-proven indicators) that lets a user know a site is “trustworthy” and produces only trustworthy information 100% of the time — even CBS News’ article on this was deemed inaccurate by the researchers.
Sadly, the researchers noted no limitations of the current study (and surprisingly, JMIR apparently didn’t require the researchers to do so before accepting their article for publication). So I’ll list them for you:
– Small sample size
– Unrepresentative sample (not randomized from the general population)
– Students in high school (e.g. results are not generalizable to the Internet population as a whole, or adults).
– Controversial health topic in the news that has likely spurred a temporary increase in controversial sites
– Search terms may have been biased (why didn’t the researchers include a more generic “vaccines” search term?)
This last point is an important one, because it occurs in research all the time. By asking the right questions, researchers can bias a study before ever collecting a single data point, ensuring the outcome they’re seeking (unconsciously or not).
Students, too, may be especially naive when it comes to understanding the complexity of health and medical information online, since few have any direct or sustained interaction with health care concerns or treatments. I would expect adults to be a little less naive and a little more skeptical of any information they find on a site that proclaims to offer “Uncensored vaccine information that the government doesn’t want you to know!”
Reference:
Kortum, P., Edwards, C. & Richards-Kortum, R. (2008). The Impact of Inaccurate Internet Health Information in a Secondary School Learning Environment. J Med Internet Res, 10(2):e17.
Well, this irks me. Who are these guys? I learned better than that in undergraduate survey courses.
Is this a respected journal? Might these authors have an agenda in doing a halfwitted study to suggest that googling is a bad idea?
JMIR is a respected journal and a leader in the field of internet/health research. Founded in 1999 when few medical journals were providing a forum for this type of research, much less cross-disciplinary work, JMIR is on the frontier and one of the first sites I turn to when I wonder, Has this research idea been explored yet? (Check out their Focus and Scope page for more info.)
However, I sometimes wish there were an “adult swim” version of the journal (and maybe there is and I am not aware of it). I’d like to see an excellence marker — which studies are especially rigorous, ground-breaking, and important? Which studies has JMIR published that would have made it into the traditional journals if those other editorial boards were better tuned in to what’s happening online?
To reiterate Susannah’s comment, yes, JMIR is a respected journal, and a pioneer in the area of publishing this kind of work as well as a pioneer/champion regarding open access. That said, the reviewers in this case simply dropped the ball. I know Gunter Eyesnbach pretty well, and his vision for the JMIR, and his own scientific, rigor are both of the “adult swim” type Susannah wishes were present. On the other hand, the weakness in publishing this article e.g. limitations not mentioned, is pretty obvious. In part, because this is still a frontier, I think they err on the side of getting these observations out there, and to allow the reader to do the critique.
Excellent post & analysis, not just uncritically reporting the results. Re: Susannah’s notion of an excellence marker or index of health information quality that consumers can trust, Joan Dzenowagis at WHO has come out for creating a top level domain of “dot health” (c.f. dot com, dot org, etc.) that would at least somewhat separate wheat from chaff (www.connected.org/is/dot-health.html). While not bulletproof, I think this has real potential!