We welcome Peter Frishauf as an author on our blog. Peter is on the Editorial Board [brief bio] of our Society’s Journal of Participatory Medicine, and as described below, has already authored some important material on this subject. His first post here is triggered by an article in Tuesday’s New York Times that generated much discussion on a vital subject: how we reality-check what we think we know, as a basis for science, and especially for medical advice. – e-Patient Dave

It’s a recitation of the obvious: without reliable, high-quality peer-review, medical information and patient care suffer.

Sadly, academia and broken peer review are also linked to our tenure system. We need to fix that.

I hate to condemn with a broad brush: many academics care deeply about better patient care and believe – genuinely – that tenure is a good thing. As Wikipedia describes it today, tenure “protects teachers and researchers when they dissent from prevailing opinion, openly disagree with authorities of any sort, or spend time on unfashionable topics.” But sadly, the vast majority of academics acknowledge that an even more powerful benefit of tenure is pretty damn good job security. Tenure has much more to do with money (lifetime job security) than academic freedom.*

Tenure and traditional peer review are joined in a pernicious union that should be smashed if universities are to be trusted to promote the public interest through good, published science and medicine. In the “publish or perish” environment of academic medicine, candidates for tenure are generally ranked not on how much information they create (by research or invention) or how well they teach (which counts for just about nothing), but by the prestige of the journal where they publish. It’s a system that might have made sense when a journal’s reputation was regarded as a proxy for an author’s scientific rigor and integrity. No more.

With the Internet, information is distributed efficiently and effortlessly. Reputations are created – and destroyed – quickly. Crowdsourcing experts review articles to improve their quality. Articles themselves aren’t published and forgotten, but continually curated over time. The old publishing and peer review model is obsolete in this environment.

In an excellent article this week in The New York Times, journalist Patricia Cohen describes a number of experiments to improve peer review:

• Katherine Rowe, a Renaissance specialist and media historian at Bryn Mawr College, worked with the prestigious 60-year-old Shakespeare Quarterly to crowdsource/review articles by a panel of invited experts. The reviewers “were invited to post their signed comments on the Web site MediaCommons, a scholarly digital network.” Others could add their thoughts as well, after registering with their own names. “In the end 41 people made more than 350 comments, many of which elicited responses from the authors. The revised essays were then reviewed by the quarterly’s editors, who made the final decision to include them in the printed journal, due out Sept. 17.”
• “Today a small vanguard of digitally adept scholars is rethinking how knowledge is understood and judged by inviting online readers to comment on books in progress, compiling journals from blog posts and sometimes successfully petitioning their universities to grant promotions and tenure on the basis of non-peer-reviewed projects.”
• “Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants. Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.”

Journalist Cohen also nails the tragic and under-reported association between traditional peer review and tenure:

• “The most daunting obstacle to opening up the process is that peer-review publishing is the path to a job and tenure, and no would-be professor wants to be the academic canary in the coal mine. The first question that Alan Galey, a junior faculty member at the University of Toronto, asked when deciding to participate in The Shakespeare Quarterly’s experiment was whether his essay would ultimately count toward tenure. “I went straight to the dean with it,” Mr. Galey said. (It would.)

Well, good for the University of Toronto: Mr. Galey can get “tenure credit” and participate in an experiment to create high quality information. What a concept for a university!

Medicine isn’t a total laggard when innovating in this space. PLOS, several journals within the BioMedCentral group, and others, are experimenting with open, pre and post publication peer review. A growing group of medical publications – but still a tiny percentage – embrace open access, which makes articles freely available. U.S. federal law that mandates open access for research published with public grant money is speeding the trend exponentially.

Still, when you consider what’s at stake – healthcare, and at times a life-saving treatment – we should hang our heads in shame when one considers how slow medical publishers have been to improve peer review and access. And shouldn’t academia be leading the charge for better information rather than obstructing it through a system as self-serving and public-interest-damaging as tenure?

In more naïve times, most of us believed that the system of peer review used by trusted, traditional publications like The New England Journal of Medicine was our best bet for trusted sources of current medical information. Now we know better: numerous experts have provided extensive evidence that traditional peer review is unreliable. And using tenure to prop up the bad system is a black eye on our universities.

I have proposed³ that online Reputation Systems could be created to help assess the reliability of crowdsourced information, medical evidence, and pre and post-publication peer review. It’s just one idea, my horse in the race. Regardless of what we wind up with, here’s hoping that everyone who wants to improve health and patient care will innovate to fix our broken peer review system. And while we’re at it, let’s fix tenure, too!

^{* When you think about it, academic freedom and tenure shouldn’t even be linked: there are other ways (transparency, public exposure, good journalism by people who write about universities, even political pressure) that influence academic freedom more powerfully than tenure.  Return to reference}

Suggested Readings, Podcasts, and Websites

1. Cohen P. Scholars test web alternative to peer review. In The New York Times. Published: August 23, 2010. Retrieved 20:38, August 25, 2010
2. Smith RW. In search of an optimal peer review system. J Participat Med. 2009(Oct);1(1):e13. Retrieved 20:40, August 25, 2010
3. Frishauf P. Reputation systems: a new vision for publishing and peer review.  J Participat Med. 2009(Oct);1(1):e13a. . Retrieved 20:42, August 25, 2010&npsp; Return to reference
4. Frishauf, P, Smith, RW, Gruman, J, Green, L. Participatory evidence: opportunities and threats.  Podcast for J Participat Med. 2010(Aug); Retrieved 20:45, August 25, 2010
5. Frishauf, P, Smith, RW, Wager, L, Jadad, A, Adler, T. Peer review and reputation systems: a discussion. Podcast for J Participat Med. 2010(Aug); Retrieved 20:51, August 25, 2010
6. Rothwell, PM, Martyn, CN. Reproducibility of peer review in clinical neuroscience. Brain 123: 9, 2000 (Sept)
7. MIT Center for Collective Intelligence. http://cci.mit.edu/
8. Tenure. (2010, August 17). In Wikipedia, The Free Encyclopedia. Retrieved 19:28, August 25, 2010, from http://en.wikipedia.org/w/index.php?title=Tenure&oldid=379372335
9. Adler TB, Alfaro, L. A content-driven reputation system for the Wikipedia [PDF]. ACM 978-1-59593-654-7/07/0005. Accessed October 17, 2009. [Google Scholar]
 The group has released a WikiTrust extension for the Firefox web browser based on its research that is now available in beta and available at https://addons.mozilla.org/en-US/firefox/addon/11087 accessed October 2, 2009. A review of the extension may be found in Wired magazine at http://www.wired.com/wiredscience/2009/08/wikitrust/ accessed October 2, 2009.
10. Priedhorsky R, Chen J, Lam SK, et al. Creating, destroying, and restoring value in Wikipedia. ACM 978-1-59593-845-9/07/0011. [Reference to come.] Accessed August 12, 2009. [Google Scholar]
11. Frishauf, P. The end of peer review and traditional publishing as we know it.  Medscape J Med. 2008;10(11):267. Retrieved 21:35, August 25, 2010