Because You Know I’m All About That Base

by · July 5, 2016

9885051386_5578035a26_k“This is a common pattern: people who have information about an individual case rarely feel the need to know the statistics of the class to which the case belongs.”
– Daniel Kahneman

I have seen some recent talk about how science has lead us too far from the particulars of the patient encounter. We have become too beholden to the general. We need to remember that the patient in front of us is unique. Not a statistic, but an individual human.

When we reduce our thinking down to statistics based on large populations, we lose sight of the individual patient. That patient is not a statistic. That patient is a unique, special thing. When considering the patient, they’re not a cold statistic, they are a human.

People love to think this way. It makes for a much better story, and humans love stories about being unique. People like to believe that they are not like everyone else; they are individual.

“I want to be different but I guess that’s nothing new.”
– Reel Big Fish

In 1975, the psychologists Nisbett and Borgida performed a now famous experiment. They explained to a group of subjects a previous experiment known as the “helping experiment”. In the helping experiment, people were asked to interact with someone in the adjacent room over an intercom, talking about their life story. In the helping experiment, one of the participants is an actor who then gives the portrayal of having a seizure, calling for help, choking, and then dying. Famously (and reproducibly) hardly anyone tries to assist him.

But that isn’t the experiment performed by Nisbett and Borgida. In their experiment from over 40 years ago, they simply explained the results of the helping experiment to half of their own subjects. The other half was unaware of the results and served as the control group. Then both groups watched two videos of people who participated in the original study. The videos presented a positive impression of the participant. Subjects were then asked whether or not they believed that this particular participant was one of the people who tried to assist the “dying” actor.

You would think that the subjects that were first educated about the results of the helping experiment would skew their responses to reflect the reality that people, in general, will not rush to assist. But they found absolutely no effect from that education. Both groups equally believed that the participants they viewed were one of people who rushed to the aid of the actor. The person in front of them was unique; different from the general rule.

To quote Nisbett and Borgida:

“Subjects’ unwillingness to deduce the particular from the general was matched only by their willingness to infer the general from the particular.”

There is a name for this kind of thinking. It is called the “base rate fallacy”. The base rate fallacy basically states that, when presented with specific details, people will completely ignore the statistical evidence explaining the general rule, and focus solely on the “unique” characteristics of the individual in front of them. It is very seductive for a reason – it is the way the human brain prefers to approach things. What is counter-intuitive is that general rules based on large samples are extremely powerful.

So what’s wrong with focusing on the individual? What if I have a really good test? Can’t I then throw out the general and find the special?

Let’s give an example. Say I just made up a new test for ACL tears – let’s call it the Meira Test because I’m a pompous idiot. Now, the Meira Test is good. Damn good. High quality research shows that it has a 99% sensitivity and a 99% specificity which means:

  • Every 100 times I do the test on people with a torn ACL, 99 times it will be positive
  • Every 100 times I do the test on people without a torn ACL, 99 times it will be negative

Now, you and I are walking down the street together. Let’s say that we are eating ice cream. (I really like ice cream. Especially coffee ice cream. I like coffee ice cream so much that when I actually drink real coffee, I dump enough cream and sugar into it to make it taste like coffee ice cream.) I turn to you and say, “Please hold this delicious cone of coffee ice cream for me.” Then I proceed to push a random stranger to the ground and perform the Meira Test on them.

The test is positive

Now, based on my 99% sensitivity and 99% specificity, how likely is it that this guy on the ground here actually has a torn ACL? Not ANY guy, but THIS guy, right here in front of me. This “N=1”. This one individual who is now calling the police to charge me with assault.

Most people would assume that the probability for this guy to have a torn ACL is 99% based on the sensitivity and specificity findings. Only a 1% chance of a false positive so that makes sense right?

Nice job! You just failed miserably

You were guilty of the base rate fallacy. But let me show you how severely off you really were with some EXTREMELY simple math.

“Come on, Meira!!! I’m not doing MATH!!!”

The base rate for ACL injuries in the general population is around 1 in 1,750. So for every 1,750 people you come across, one will have suffered an ACL tear. So 0.06% of the general population. That means if we looked at 1,000,000 people, we would find 600 ACL tears. Of course that would also mean there would be 999,400 people without an ACL tear.

  • When I do the Meira Test on the 600 people with an ACL tear, I get 594 positives (99% of them)
  • When I do the Meira Test on the 999,400 people without an ACL tear, I get 9,994 positives (1% of them)
  • So we have a total of 10,588 positives (594 + 9,994 = 10,588) which means 594 of 10,588 positives actually has a torn ACL or 5.6% (594/10,588).

Now, I used my test with a 99% sensitivity and 99% specificity (damn near perfect test) but the probability of THIS positive finding actually being a torn ACL is just 5.6%. In other words, the probability of this guy NOT having a torn ACL is 94.4%.

(If you care, what I did there was apply Bayes’ Theorem using objective priors.)

In other words, we could learn a lot from Meghan Trainor (she misspelled “base”)

Narrowing down populations

Here is where more analysis is helpful. If the guy I tackled is an adolescent/young adult then the probability of him having an ACL tear after performing the Meira Test jumps to 37.8%. I know this because that population has a 1 in 175 incidence rate for ACL tear. But it is still much more likely that he did NOT tear his ACL.

But what if he was also a female soccer player who planted and pivoted when she felt a “pop” and gross instability? As I narrow the population down further and further my confidence in what I know increases. But this is based on published data collections.

I didn’t just say, “Yeah, well, he lacks great toe mobility which, in my experience, is VERY important so I’m going to weight it super heavily, throw out the base rate entirely, and say he likely has a torn ACL.” (Expert opinion, or “subjective priors”, plays a big role in Bayesian analysis but that is VERY specific in its application – you should read this piece by James Berger)

What is often argued is that as you narrow the population down to an individual, or “N=1”, ANYTHING is possible or even likely! It is one thing to argue this in order to avoid missing something potentially fatal (even then, it is still not recommended to throw out the base rate). But it is most commonly used in physical therapy practice to support an intervention that has been shown not to be very effective across a large population of patients. When you do this, by definition you neglect the vast majority of your patients.

Besides, if every patient is so exceptional and likely to respond to that individualized-reasoned plan of care, why is it that studies have not found multimodal individually tailored treatments to perform any better?

On the other hand, at least you get to keep from changing the way you practice. And maybe you can come off as someone’s hero in the short term…

In summary…

  • We can learn a lot from Meghan Trainor
  • Humans think that the specific outweighs the general
  • Narrowing down a population is not arbitrary
  • Don’t neglect the vast majority in the name of finding outliers
  • Coffee ice cream is tasty

The featured image on this post is “Base” by Roel Wijnants via Flicker.