Dr. Robert Wachter has an interesting essay over at THCB entitled, Should Patient Satisfaction Scores Be Adjusted for Where Patients Shop? As health care in the U.S. continues to move in the direction of tailoring itself to patient satisfaction, the question becomes — how do we make such ratings more reliable and fair? The answer is, “Not easily.”
Patient satisfaction in a hospital is a many-varied thing. You could have the rudest doctor in the world (ala TV’s “Dr. House”), but if he ends up saving your life, how badly could you rate the rest of the hospital? You could end up dying of a preventable infection from the hospital, and yet, who’s going to capture your unhappiness once you’re in the grave? You could have the nicest, most kindest surgeon in the world, but if he leaves a sponge in you and has to re-open you back up to retrieve it (but tells you it’s just “a routine check to ensure everything is healing properly”), is your hospital rating really going to help others understand that hospital may not be the best one to visit?
Wachter’s point is a valid one — that we need to adjust hospital ratings, and even ratings within the same hospital, based upon the patient’s experience. Outpatient is usually a very different experience than inpatient. Staying in a psychiatric bed is usually nothing like staying in the ICU. Dealing with the madness of the E.R. is nothing like going in overnight for a routine colonoscopy. A hospital in a poor urban area is generally going to be more poorly rated than one in a rich suburb.
Researchers have long recognized the importance of rating “apples to apples.” If you try and compare the efficacy of an antidepressant, for instance, with say, an asthma inhaler, I’m sure you’ll get two very different results. But they shouldn’t have been compared in the first place, because they have virtually nothing to do with one another.
The same is true as we experiment with new ways to provide consumers with more information about the hospitals in their community. These rating systems should be carefully and scientifically devised, normed, validated, and then used only for “apples to apples” comparisons.
And the same can be emphatically said, too, for online ratings of virtually anything. Almost no online ratings’ systems have been normed, empirically validated, or have any sort of random selection occurring. This means that virtually every rating system you come across online — whether it’s for medications, or doctors, or, well, even a TV — is not really a scientific measure on which you should be basing your decision. The population of people submitting these ratings are not a random sample, and so what you see and read online should be taken with a grain of salt.
Eventually, all of these things will be sorted out and we’ll find some happy medium. Until then, we’re left with a lot of pseudo-science and questionable data.
Thanks, John, this is really of the moment for me since our city glossy just produced their annual “Top Doctors” list (which you can’t view online, but you can read how they come up with their list (a survey of doctors, no consumer input).
Since I know something about a few specialties here in town (thanks to personal experience and DC Urban Moms) I turned right to those sections and was disappointed to see a doctor listed who is on the underground “Do Not Call” list. Like seeing erroneous info on Wikipedia, it calls the whole enterprise into question.
Another reason I’m skeptical: I worked for U.S. News & World Report 1995-2000 and witnessed the sausage factory that is the college rankings process.
I thought a 2001 article by a former director of data research at U.S. News blew the whole thing wide open, but the rankings march on like a zombie, haunting high school seniors and their parents, even while smart people beat the drum against them.
I agree that online ratings are pseudo-science, as are these annual old-media rankings, but I do think (hope?) that with more consumer input, they can be improved.
> Should scores be adjusted for where people shop
Synchronicity: last week a hospital near me posted a new billboard that almost literally says “We’re rated #1 by our patients!”