Base Rates and Biases

Hey everyone, and welcome back to The Long Run.

Quick programming note: I’ll be off next week on vacation, so this newsletter will pick back up in 2 weeks.

If you like this article, you can subscribe below!

You can also check out my previous posts here and follow me on LinkedIn or Twitter.

Let’s hop into it.

There’s an interesting paradox in how we interact with each other. We’re genetically predisposed to fit in, and we do our best to assimilate with the group around us. Yet, at the same time, we have a drive (almost a need) to be different — to express ourselves as individuals.

We are all the same, yet we’re all different.

Particularly in America and in Western cultures, this desire to be different — to be exceptional — has probably served us well and led to many of the greatest breakthroughs in technology, industry, art, music, sports, and just about everything that we do.

But inherent in the belief that we are each uniquely different from anyone else comes the idea that averages don’t apply to us.

The representativeness heuristic

In psychology, there’s a well-known concept called the representativeness heuristic, which states that we estimate the probability of an event based on how similar it is to a situation we know.

As an example, there’s this guy sitting at the next table over from me in the coffee shop who is here every week. He dresses in crazy outfits, walks around barefoot sometimes, and appears to be aggressively jotting down words into his notebook.

From everything I’ve told you about this guy, would you be more likely to guess that he’s a playwright or an engineer?

Obviously this eccentric guy in an LA coffee shop would be a playwright, right? Maybe. But in reality there’s something like 2 million engineers in the US, yet only around 50,000 writers.

So numbers would tell us that the average random person we walk into anywhere in the US is 40x more likely to be an engineer than a playwright. However, we’re quick to create stories about people based on the limited information we have.

When we make guesses, we often forget to consider base rates. Too often we fail to look at base rates when making decisions, estimates, or judgments. We assume that we know better, or “that doesn’t apply to me.”

The closer we look, the more we see this phenomenon pop up everywhere — across our social lives, our financial decisions, and even in our health.

Base rates in our social lives

Are you married? If so, do you have a prenup? Prenuptial agreements are only used in about 15% of marriages, yet nearly 50% of marriages end in divorce.

While maybe having a prenup is viewed as a sign of bad faith, the numbers don’t lie. And yet, we hear these statistics and think, “oh, that doesn’t apply to me.” Of course we believe our love will stand the test of time — if we didn’t, we wouldn’t be getting married in the first place.

Base rates in markets

A similar concept arises in investing, particularly in risky markets. As an example, in the stock market, IPOs tend to “pop” in price on the first day, as everyone is excited and eager to get in on the next big thing. However, that enthusiasm typically wanes, and most companies post-IPO tend to underperform over the long run, with 64% lagging the broader market by over 10%.

But the ones that outperform do so by such a large magnitude that we hear those stories and want to believe that every company has a shot of being the next grand slam.

Home runs make the news, while we never hear about the thousands of mundane groundouts to second base.

Base rates in our health

The same concept applies to our behavior and the actions we take that impact our health. Statistics would tell us that by nearly any measure, smoking is bad for us. Yet many smokers will point to an example of a person they know that lived to be 100 and smoked all their life.

They ignore the averages and instead point to outliers in favor of a narrative they want to believe.

People love to tell the story of how Warren Buffett eats McDonald’s and drinks Coke every day. “And look at him! He’s almost 100 years old and is still running one of the best investment firms in the world.”

Sure, the fact that Warren Buffett likes junk food, the fact that he has lived a long life, and the fact that he is one of the greatest investors of all-time, all may be true in isolation. But it would be crazy to believe that his diet is what enabled his career success and longevity. Data would show that, on average, McDonald’s and Coke are probably net-negative for our health when consumed regularly.

Survivorship bias and understanding historical data

Survivorship bias is a logical error that distorts our understanding of the world, and it happens when we assume that the successes tell the whole story, but we don’t adequately consider past failures. For every success, there are often thousands or millions of failures.

For every unicorn startup, there are tens of thousands that fizzled out. For every person who smoked and lived to 100, there are millions who died much earlier.

When evaluating any historical data, we need to look at the averages, not just the exceptional outliers, to properly assess true probabilities.

One of the best parts about studying history is to see what has actually worked well and what turned out to be a disaster — we can learn from both.

But we must also remember that history only works as well as we’re able to accurately view its records. And since most data on the economy and financial markets is from the last ~100 years, pure numbers won’t give us the full picture. It’s hard to accurately predict how markets will respond to increasingly complex macro and environmental conditions the likes of which we have no data on.

Yes, history is cyclical. We’ve seen several market cycles over the past few decades, but we haven’t seen nearly as many generational cycles, even less regime cycles, and almost no complete changes of empires or shifts in major world powers.

Rising interest rates, inflation, record low unemployment, peak housing prices, and a tech hype cycle around AI, all in the aftermath of a global pandemic? We don’t really have data that would predict where we go from here.

And I think that makes it all the more exciting. We have to lean even more on narratives and human intuition when we don’t have data or historical precedents.

Understanding base rates, yet choosing to deliberately disregard them

So we can take all of this knowledge and be more aware of how base rates play a role in our everyday lives, but in reality, many of us will continue to disregard them.

The idea that we are personally immune to averages, that we are exceptional, is a large part of what it means to be American. Without it, we would never take big swings. We’d live in our comfort zones and just stick with the status quo. While this may be the correct path to a safe and secure life, so many of the treasures in life come from committing to something deeply under the (probably irrational) belief that we are better than average.

Common strategies yield common results, but there’s a less quantifiable thrill — call it “the process,” call it experience, call it the journey along the way — that isn’t a purely measurable outcome. And I think it’s precisely those immeasurable things — the hopes, dreams, and stories created along the way — that just might justify knowing the base rates, yet choosing to act against them in certain situations.

Personally, I chose to work at a startup, despite knowing that over the long run 90% of startups fail.

Why? Because I had enough conviction to believe that this specific company was part of the 10% that survive, and heck, even the .01% that become unicorns. From a purely probabilistic and expected value calculation, this is highly irrational. I’d be better off getting a safe corporate job.

But this is where the immeasurables come into play. The rate of learning, the ability to work closely with a team of incredible people, the stories created along the way, holding onto the belief that we have a chance at outlier success — you can’t put numbers on these things, but they make our experiences distinct. They give us meaning, a purpose, a reason to wake up in the morning.

As we wrote about before, converging to the average is typically the safe bet.

But what kind of Americans would we be if we were okay with average?

-Owen

Fresh Finds

On the Debt Ceiling, the White House Is Doing What It Said It Wouldn’t Do (Odd Lots)

Podcast | 43 minutes

For those of you curious what the heck is going on with this “debt crisis” that’s been floating around the news, this episode interviewed Skanda Amarnath and Arnab Datta — two people who understand it better than almost anyone else. Thought this was an excellent primer on the current situation we face and if it could have been avoided (spoiler alert: it probably could have been).

If you enjoyed this piece, subscribe here to receive the newsletter! Once a week, no BS, and sure to make you think.

Also, if you’re interested in starting your own blog or newsletter, check out Beehiiv here!