We are showing a big push in evidence based practice throughout healthcare, meaning not only are we doing new research to try and determine what new exciting treatments we can use for various conditions, but we are also taking a look back at “classic” treatments to determine if they stand up to modern rigors. But, beyond just the existence of a study, quality of research is a GARGANTUAN component in determining if a treatment is evidence based. So, how do you as a consumer analyze to see if a publication is worthy of your consideration? How do you know when “research shows” is meaningful?
The first thing we should look at is the type of study we are reading. Lowest on the list are what we call “case reports.” These are pieces written by a clinician regarding the treatment they did on one or just a couple of cases of an issue. The problem with this is that it is a retrospective look at a treatment that cannot isolate out other variables, and one instance of a case is at high risk for abnormal results, meaning we are unable to apply statistical analysis to how it would work on the majority of patients. Take for example that one person who wins the lottery and their good luck rituals beforehand. Just because it worked for them that one case does not necessarily mean it will work for them again, or for you in the future. However, it may be worth investigating further with better research, which brings us too…
Cohort studies! These studies look at a treatment for an issue given to a much larger group of persons to determine if we have a statistically meaningful effect on that population. Now, how effective this is compared to other treatments and whether or not it compares to a placebo is still not analyzed, but we can at least see if that treatment has any possibility of moving forward in the process of testing. Again, good utility for researchers, but not great clinical value yet.
So now we get to the big important stage: randomized, controlled trials (RCTs). These trials take data from previous research stages and control for other variables in treatment, time, placebo effect, comparison to other successful treatments for superiority vs inferiority of treatment (not just does it work, but does it work better or worse than what we’re already doing), and (hopefully) perform them on a large enough population to be statistically meaningful. If we have that large enough population, we call this a “well powered” study, meaning there are enough people involved in the trials that we won’t accidentally rule in or out something key. A positive result at this stage is huge, because it lends a lot of support to the treatment being tested. With that said, errors can still be made along the way, so other researchers will retest the theory in other experiments to determine if the results regularly stand up. That brings us to the big fish, the gold standard, and really the only research you as a consumer need to care about…
Systematic Reviews and Meta Analysis! In this paper, the researchers are going to go through a rigorous algorithm to compare the quality of the RCTs we just talked about to determine if any of them were at risk for major errors, if they have results that are well interpreted and meaningful, and if there was any risk for bias in how the study was conducted or written that may have altered the results. A good result from one of these studies is the gold standard for whether or not a treatment should be used!
Now, we just talked about something important here: bias. Just like everyone else, researchers are human. This means that there is risk of financial interest (so we watch carefully for studies that were funded by a corporation that created the treatment because they can benefit financially from a positive result), confirmation bias (the researcher really wanted this to work, so they unconsciously influenced the study accidentally), or just blatant falsifying of data (the researcher wanted a particular outcome for one reason or another so badly that they actually lied about the results). So, some things we look at to try and limit that risk include blinding assessors (they don’t get to look at if the results represent a person in the treatment group or not), using external statisticians unconnected with the research team or company, the peer review process through reputable journals to ensure methods line up appropriately, and ensuring funding and conflicts of interests have been declared. A good systematic review will look into all of these things before running their analysis!
So, what does this mean for you? It means research is amazing, but there’s a lot to look at! Be picky and analytical about study findings, know the type of study you’re reading, and ask your doctors questions if you read something and aren’t sure! This can keep us on the forefront without falling for any snake oil!
