Bayes theorem, the geometry of changing beliefs

The goal is for you to come away from this video understanding one

of the most important formulas in all of probability, Bayes' theorem.

This formula is central to scientific discovery,

it's a core tool in machine learning and AI, and it's even been used for treasure

hunting, when in the 1980s a small team led by Tommy Thompson,

and I'm not making up that name, used Bayesian search tactics to help uncover a

ship that had sunk a century and a half earlier,

and the ship was carrying what in today's terms amounts to $700 million worth of gold.

So it's a formula worth understanding, but of course there

are multiple different levels of possible understanding.

At the simplest there's just knowing what each one of the parts means,

so that you can plug in numbers.

Then there's understanding why it's true, and later I'm going to show you a

certain diagram that's helpful for rediscovering this formula on the fly as needed.

But maybe the most important level is being able to recognize when you need to use it.

And with the goal of gaining a deeper understanding,

you and I are going to tackle these in reverse order.

So before dissecting the formula or explaining the visual that makes it obvious,

I'd like to tell you about a man named Steve.

Listen carefully now.

Steve is very shy and withdrawn, invariably helpful but

with very little interest in people or the world of reality.

A meek and tidy soul, he has a need for order and structure, and a passion for detail.

Which of the following do you find more likely?

Steve is a librarian, or Steve is a farmer?

Some of you may recognize this as an example from a study

conducted by the two psychologists Daniel Kahneman and Amos Tversky.

Their work was a big deal, it won a Nobel Prize,

and it's been popularized many times over in books like Kahneman's Thinking Fast and

Slow, or Michael Lewis's The Undoing Project.

What they researched was human judgments, with a frequent focus on when these

judgments irrationally contradict what the laws of probability suggest they should be.

The example with Steve, our maybe-librarian-maybe-farmer,

illustrates one specific type of irrationality,

or maybe I should say alleged irrationality, there are people who debate the

conclusion here, but more on all of that later on.

According to Kahneman and Tversky, after people are given this description

of Steve as a meek and tidy soul, most say he's more likely to be a librarian.

After all, these traits line up better with the

stereotypical view of a librarian than a farmer.

And according to Kahneman and Tversky, this is irrational.

The point is not whether people hold correct or biased views about the

personalities of librarians and farmers, it's that almost nobody thinks to

incorporate information about the ratio of farmers to librarians in their judgments.

In their paper, Kahneman and Tversky said that in the US that ratio is about 20 to 1.

The numbers I could find today put that much higher,

but let's stick with the 20 to 1 number, since it's a little easier to illustrate

and proves the point as well.

To be clear, anyone who has asked this question is not expected to have perfect