Cochrane Collaboration logo – it’s a forest plot!
If you’ve been paying attention, you know that yours truly got the chance to attend the annual Cochrane Colloquium in Edinburgh in September this year, thanks to a travel stipend from SPM, a #PatientsIncluded bursary from Cochrane UK, the hosts of the 2018 Colloquium, and a stipend from the NHS for the #BeyondTheRoom project to help cover the event for the global audience.
And I’m sure there are a good number of you who are still thinking, “That’s great, but what the heck is Cochrane? And how did they get to be the ones running the ‘Hogwarts Sorting Hat’ of global medical evidence?” Forgive me, but I’m a comedy writer, and that breaks through from time to time, no matter how hard I try to stifle it. Anyway, Cochrane is named for a Scottish doctor, Archibald Leman “Archie” Cochrane, who wrote “Effectiveness and Efficiency: Random Reflections on Health Services” (the link will let you download the whole book in PDF) in 1972.
Archie Cochrane advocated for randomized clinical trials (RCTs) for, well, everything – treatments, practice methods, research protocols, an “all of the things” approach, on a loop – which was not how medicine was being practiced under the prevailing “doctor knows best” practice model in place across the globe. “The art of medicine to preserve autonomy, the science of medicine to preserve authority” rules pointed out by many people seeking to make medical science more science than “because it’s how I do things” – those rules have been snarked at by both your correspondent, and Dr. Al Mulley at Dartmouth, among a host of others.
Archie Cochrane influenced the thinking, and practice, of many other clinicians with his thought leadership on practice variation, practice standardization, and the use of RCTs to fine tune medical science. One of the people he influenced was Iain Chalmers, who, in 1993, founded the Cochrane Collaboration in Archie Cochrane’s memory. Here’s a graf from “A brief history of Cochrane”:
“The Cochrane Collaboration was founded in 1993, a year after the establishment of the UK Cochrane Centre in Oxford, UK. The UK Cochrane Centre arose from a vision to extend a ground-breaking programme of work by Iain Chalmers and colleagues in the area of pregnancy and childbirth to the rest of health care. Inspired by Archie Cochrane’s claim that “It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials” (Cochrane 1979), Chalmers and colleagues developed the Oxford Database of Perinatal Trials and a series of systematic reviews published in Effective Care in Pregnancy and Childbirth (Chalmers 1989). The database became a regularly updated electronic publication in 1989, developed into Cochrane Pregnancy and Childbirth Database in early 1993, and formed the basis of the broader Cochrane Database of Systematic Reviews (CDSR), launched in 1995. Work on a handbook to support authors of Cochrane Reviews had begun in 1993, and the first version was published in May 1994. Over its first 20 years, Cochrane has grown from an initial group of 77 people from nine countries who met at the first Cochrane Colloquium in Oxford in 1993 to over 31,000 contributors from more than 120 countries in 2015, making it the largest organization involved in this kind of work (Allen 2006; Allen 2007; Allen 2011). Cochrane is now an internationally renowned initiative (Clarke 2005; Green 2005).”
Since 1993 – only 25 years – Cochrane has spread across the globe, with centers on every populated continent:
Cochrane UK (and their Evidently Cochrane blog, which is terrific)
Cochrane Australia (also supporting emerging networks in Indonesia and the Philippines)
Cochrane Chile (hosting the Cochrane Colloquium global meeting in 2019 in Santiago)
The above list is just a sampling – and you may notice that there’s somebody missing. Yes, I’m looking at you, USA. The US did have a Cochrane Center home based at Johns Hopkins in Baltimore, but that closed in February 2018. For now, the best we’ve got is the Cochrane US West Center at Oregon Health and Science University in Portland, Oregon. But that’s a story for another blog post.
On the e-patient front, Cochrane has some terrific stuff on tap. They have a vibrant global consumer presence, via the Cochrane Consumer Network, and a ground-breaking new global citizen science project, Cochrane Crowd, where anyone can take part in the research synthesis process. The Crowd platform provides all the training anyone might need to be able to participate in assessing RCTs and studies, after completing it you’ll be ready to go, sifting through studies and trials to separate the good science from the questionable and not-reproducible stuff.
Cochrane popped up on my radar screen sometime in the last decade or so, during the time that I was scrambling to get on top of managing my parents’ care in the last few years of their lives. It came in handy as I was sifting through my decision tree during cancer treatment ten years ago, and as I’ve become more and more interested in killing off quackery and over-, under-, and mis-treatment in medicine in my work as a citizen science activist and ground-level health policy wonk. If you’re interested in the same things, join the party. We’re all in this together, and Cochrane can help us move the needle toward what I call “Goldilocks medicine” – the right treatment for the right patient, at the right time – at a faster rate.
Editor’s update: the final post in the series is Whither Cochrane, for e-patients and everyone else?
Cochrane is the most Democratic organezation,that involves patients and researchers working together as partners,that i ever met.
Bravo Casey! Excellent historical summary and open call to the USA to step up with their opening of a new Cochrane Center.
Cochrane analyses are terrific and have added a lot to our understanding but are also flawed. They assume that in any trial, all the participants have have the same condition and are aiming for the same outcome. For example all have arthritis and the outcome of interest is pain (the population is homogeneous).
However, in many community interventions such as those I have run for many years the aim is to serve populations where people come with different conditions, arthritis, diabetes, heart disease, (the population is heterogeneous) and the outcomes may be different for different people, some may never have pain so cannot reduce it but do have fatigue or depression which can be helped. The problem is that while most of the people “get better” they do so with different outcomes.
This means that the effect for each outcome is muted and thus the whole intervention looks less effective than it really is.
I could go into the statistics that are used by Cochrane but will make your eyes roll. All of this is to say that Cochrane techniques are great but have limitations that are often not well understood and this flaw affects policy and ultimately may deny patients useful, low cost, interventions.
Hi Kate – great to hear from you again. I have a question.
I think I understand your point, which may be inherently baked into the whole Cochrane approach. As I understand it, a major problem of observers looking at any one study is that other studies might not have reached at all the same conclusion, for any of a thousand reasons, even if each study is performed well. So, as I understand it, Cochrane does the extra work of looking at many studies to see what results seem to be consistent and thus more reliable, in principle at least.
Am I right so far? (Post author Casey, your two cents two?)
But inherent in that is a problem I (and others) have pointed to elsewhere: what if (a) they weren’t all actually measuring the same things, and/or (b) those things weren’t equally important to all patients in the first place?
I think that’s what you’re referring to here:
> This means that the effect for each outcome is muted and thus the whole intervention looks less effective than it really is.
“effect for each outcome is muted” means all the different potential benefits get mushed together into one evaluation that supposedly applies to all patients, yes?
But for “the whole intervention looks less effective than it is” – are you talking about just one intervention, and saying its benefit will vary between patients? Or, what part of that spectrum of benefits are you talking about?
Thanks for taking time to share your experience.
(For those who don’t know, SPM member Kate is one of the true pioneers of the field. Doc Tom Ferguson’s white paper cites her work twice – in chapter 2 (the foundation of the whole paper) and near the end, in Chapter 7, “The Autonomous Patient.”)
Have followed along with this comment thread so far – my POV here is very much from the angle of the common-human-on-the-ground vs. statistical modeling expert or statistician. My purpose in putting Cochrane on the dashboard of e-patients and other citizen scientists is to primarily expose folx to the fact that the organization exists.
F’rinstance, there are people who are leaders in patient communities who had not been familiar with what Cochrane is about beyond “it’s a library of studies.” Which it is, but there’s a lot running beneath that library. As Kate points out, how that library’s virtual shelves are filled is an imperfect process.
My thinking here is that by getting a wide array of e-patients and other interested parties at the ground level involved in working with researchers to make the science of medicine more visible and transparent to the common human, while also trying to improve the methods used to parse what works, what might work, and what doesn’t work, could actually improve that science.
I recognize that the concepts of precision medicine and setting solid treatment guidelines and protocols are cognitive dissonance on the hoof, but I’d like to see work progress on both sides of that equation, with patients included. If the process isn’t perfect, let’s work on improving that process. If the numbers don’t add up, call that out.
I don’t know of any other global effort at tackling the conundrums presented by trying to making the medical science of human health uniform, in the face of our species’ infinite variation. Getting that infinite variation involved in trying to figure that out seems like a good idea to me. Or I could be all Don Quixote up in here.
I agree but also add caution. Cochrane is the basis for most evidence based medicine and the policy developed around evidence based medicine. For many things it is excellent and has moved us forward. However, for some types of research, especially those aiming at population health, the methodology is flawed, and the flaws are not widely recognized. As a result meta analyses using Cochrane methodology sometimes lead to the discounting of effective interventions. BTW I am also a major consumer of health care having been born with a chronic condition and also being a cancer survivor.
I am all about the caution, too. I have a rep for biting all the hands, whether they’re feeding me or not, in service of moving medical science in a more patient-primary, patient-autonomous direction. And I know that Cochrane’s influence can sometimes mean that questionable “rules” get set, which are hard to un-set once in place. That precision/population conundrum guarantees tension between what an individual patient might want, or discover works for them, even in the face of guidelines and standards of care set by The Experts. Getting all (or most) of us involved in setting the rules for All of Us might help make that tension less … tense.
You almost have it but not quite. I will be a bit technical but give an example.
Cochrane makes recommendations based on effect sizes. These are calculated as the difference in the mean changes (between control and treatment groups) at the end of the study divided by the mean standard deviation of the two groups at baseline.
So let us take pain as an example. Given a ten point scale with 10 being extreme pain and 0 being no pain the two groups at baseline had an average (mean) pain of 5 with a mean standard deviation of 1.5.
At follow up the mean pain for the treatment group was 3 and for the control group 6 so the difference was 3 divided by 1.5 and then the effect size is .2 which is a small effect size.
In this case the population was very heterogeneous with some folks having lots of pain and others not having much pain. I know this because the standard deviation is fairly large. This is what happens in community studies. Let us say that that the population was chosen all to have similar amounts of pain in which case the standard deviation is only .75 which would make the effect .4 which is much larger and more meaningful.
Researchers can manipulate the effect sizes by having a very narrow range of people entered into a study. You can see what they did in studies by looking at the inclusion criteria. For example a paper might say that to enter a study people must have a pain level of 4 or more.
Now there is a second problem. Let us say that not only was the population very heterogeneous but that different people had different problems. Some people had pain and some had fatigue and some had both. Not only do you have a large baseline standard deviation, but you do not have any chance of seeing improvement in any of the people who did not have the problem at baseline.
So let’s say that of 100 people, 50 had pain and 50 had fatigue. The controls increased their pain by .5 and the treatment group reduced their pain by 1 for a 1.5 difference and the standard deviation is 1.5. This means the effect size is .1 which is not much. The intervention does not look great. However, if you only looked at the outcomes for those with pain (something you cannot do as a primary outcome in a randomized trial) then the controls increased their pain by 1 and the treatment group reduced their pain by 2 and the standard deviation was .75 so for those with pain the effect size was .4. Same intervention, same people, the only difference was excluding those who never had pain from the calculation.
Because of this problem, all our recent studies have reported the primary outcomes for the whole population along with effect sizes and then we have reported the effect sizes for those that ever had the specific problem.
I know, much more than you wanted but then you did ask.
I have a feeling there’s a mountain of insight to be gained from this discussion, for those of us who are “kinda familiar” with the subject of “effect size.” I’ll say a little about it for those who are completely new to the subject. (Kate’s comment here led me to do some googling … it was easy to find huge pages of grey text about effect size, but not so easy to find an “aha” page.)
I knew of the general idea – a study may show that their intervention (a drug, an environment change, whatever) did move things, but how much? How much of an “effect” did it have? And lots of things can make that a dicey question.
After a couple of hours of amateur studying, I think the best short answer is this graphic, taken from a ScienceDirect article.
The graphic shows two very different populations in a study – the one on the left is a bunch of people who are very similar; the one on the right has a lot of variation. In each case, imagine that the peak on the left (solid line) is where the group sat before the study, and the peak on the right (dotted line) is where they sat when the study finished. In both cases, the whole group moved the same amount. But which change is more significant?
Well, look – on the right side there’s a ton of overlap – most people fit inside both curves, and on the left side only a few people fit inside both. I’m sure I’m cheating a bit on the precise definition, but basically that means the one on the left has a bigger “effect size.”
To return to a key point that Kate made, a researcher can make his/her study look more impressive by choosing a population with very little variation, like the one on the left. That way, any movement that happens in the study will get a bigger “effect size” score than if the starting population were more diverse.
And if I understand correctly, here’s the problem with that: in reality, real populations of people are diverse – they aren’t like the curve on the left. People are very different, so a study that “cherry-picks” its subjects like that isn’t helping us much in improving the REAL world.
Kate, given my limited interpretation, is that right, as far as it goes?
The above is a simplification. The actual calculations are so hairy that the Wikipedia page on Effect Size starts by saying
“This article needs attention from an expert in statistics. (May 2011)”
That was 7 years ago. And then 3 years later someone added that it’s already TOO technical:
“This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details.”
Ah yes: please make it simpler, but don’t remove the technical stuff! Science can get complicated… that’s why (IMO) it’s essential to get feedback from people who work in the real world, e.g. the community studies Kate mentions doing.