Steaming Pile of Research
Dog conducting a “Treatment A” compared to “Treatment A + Treatment B” study.
There is a particular type of study that I really, really hate. It is extremely common and often seen in the physical therapy research. I see it cited time and again by speakers at conferences and used as a defense of a lot of interventions. That study design?
“Treatment A” compared to “Treatment A + Treatment B”
You’ve seen it. “We know that Treatment A is effective, but what if we add Treatment B?” Run an RCT. “Oh look! Even MORE effective! Make sure to add Treatment B! Only a devil worshipping satanist would not provide Treatment B!” (As a devil worshipping satanist I find this offensive.)
In the medical literature, Treatment A is usually “Standard Medical Care” while Treatment B is some kind of alternative medicine or other treatment with an unknown mechanism or efficacy. In the physical therapy literature, Treatment A is usually exercise while Treatment B is usually some kind of…well…something else we don’t need to call out. (It rhymes with “shanual sherapy”.)
This kind of study design leaves us with nothing more than a steaming pile of crap
Please researchers, stop doing this kind of study. Please educators, stop citing this kind of study. Please clinicians, stop using this kind of study to defend what you do. And the rest of you? LOOK AT THE SHINY OBJECT!
Large Shiny Object
The study design is useless because it is inherently flawed. The problem with these studies is that we usually don’t have a great idea of what exactly Treatment B does. We just know that “It works”. Because of this, you don’t know if Treatment B did anything specific. In other words, the study could actually be making this comparison:
“Treatment A” compared to “Treatment A + Placebo”
Now there’s a surprise! Adding a placebo to an already effective treatment improves outcomes. Groundbreaking stuff right there…
But the only difference between the two groups was Treatment B
No. It wasn’t. Receiving Treatment B means receiving more time, more attention, more face validity. All the things that enhance the non-specific effects of BOTH treatments. There is a lot more going on than just Treatment B. To be fair, it doesn’t necessarily say that Treatment B is ineffective. The problem is that the study doesn’t really say anything at all. Keep in mind that most treatments of unknown efficacy can NEVER be proven not to work. (A good default position is to stay away from those treatments altogether but you can do whatever you want.)
A better study design would compare “A+B” to “A+C” so at least the two groups are closer in relation. Then you can draw a comparison between Treatment B and Treatment C. That said, you STILL may be comparing the placebo effects of one to the other.
Wait a minute you jerk! Don’t we WANT to maximize outcomes?
Yes, but that kind of study design does not SPECIFICALLY support the use of Treatment B to maximize outcomes. By definition, a placebo is not specific. That means that you can use ANY placebo in place of Treatment B to maximize outcomes.
SWEET! That means I can use any treatment I want! Whatever works for the patient in front of me!
Dammit, internet! Why are you so fucking stupid? That is the same as providing a sugar pill, faith healing, or Reiki. A placebo is not something that you give in isolation. It is something that you layer on top of your intervention of known effectiveness.
In other words, you add placebo in the delivery of Treatment A. You weave it into the treatment itself. How? By becoming a master of Placebo-Fu.
In Summary…
- Not all RCTs are created equal
- Adding a treatment of unknown mechanism or efficacy could be adding nothing more than a placebo
- Some internet users are stupid
- By all means add a placebo, but add it to an effective treatment
The featured image on this post is Dog Pooping by Indi Samarajiva via Flickr.
The shiny object is Big shiny thing by Jason Rogers via Flickr.
Looking for the comments section? Learn why you can’t find them.