The Law of Large Numbers is one of the most frequently...

@TivadarDanka
Tivadar Danka@TivadarDanka
21 views Feb 12, 2025 ~3 min read
Advertisement
1
The Law of Large Numbers is one of the most frequently misunderstood concepts of probability and statistics.

Just because you lost ten blackjack games in a row, it doesn’t mean that you’ll be more likely to be lucky next time.

What is the law of large numbers, then?
Media image
2
The strength of probability theory lies in its ability to translate complex random phenomena into coin tosses, dice rolls, and other simple experiments.

So, let’s stick with coin tossing. What will the average number of heads be if we toss a coin, say, a thousand times?
3
To mathematically formalize this question, we’ll need random variables.

Tossing a fair coin is described by the Bernoulli distribution, so let X₁, X₂, … be such independent and identically distributed random variables.
Media image
4
By definition, our i-th Bernoulli random variable is 1 if the i-th toss comes up heads; 0, if it comes up tails.
Media image
5
What we are interested in is the behavior of the so-called sample average.
Media image
6
We can simulate the coin tosses in Python.

Here is the result, plotted by the evergreen Matplotlib.
Media image
7
(If you want to do some simulations yourself, here is a Google Colab notebook which I used: colab.research.google.com/drive/1Wo74sRn…)
8
As you can see, the sample average gets suspiciously close to 1/2 as the number of attempts grows.

This value is not just the probability of heads, but also its expected value.
9
So, we conjecture that the sample averages converge to the expected value.

Can we conclude this from our simulation? Definitely not. Notice that the sample average is a random variable, so we might have been just lucky. Let’s toss even more coins to find that out.
10
In the case of coin tossing, we can explicitly calculate the distribution of sample averages. As a single coin toss is Bernoulli(1/2)-distributed, their sum has a binomial distribution.
Media image
11
What does this distribution look like? Is it concentrated around 1/2? Let’s do more simulations to find out.

Here is the (empirical) distribution of the sample averages, once more visualized by the good old Matplotlib.
Media image
12
We can see that as the number of tosses grows, the distribution of the sample average becomes more and more concentrated around 1/2.

How can we express this in a mathematically precise way? Enter the weak law of large numbers.
13
To study the general case, let X₁, X₂, … be independent and identically distributed random variables.

The probability of the sample average falling farther than ε from the expected value μ is expressed by the following.
Media image
14
The so-called weak version of the law of large numbers describes that these probabilities behave as we expect.
Media image
15
However, this doesn’t mean that we were not simply lucky with our first simulation.

Can the sample average converge to a number other than 1/2? In principle, yes. Think about the extremely unlikely case of only tossing tails, resulting in the sample average converging to zero.
16
Even though tossing only tails has zero probability, are there other similar outlier cases, perhaps happening with a nonzero probability?

What is the probability that the sample averages converge to the expected value? The strong law of large numbers gives the answer.
Media image
17
Thus, a losing streak in any game of chance doesn't mean that you'll get lucky soon.

The law of large numbers cannot be applied in this context: it applies only in the long run. Anything can happen in the short run.
18
The law of large numbers is also the topic of the latest chapter of my Mathematics of Machine Learning book.

Check it out if you are interested in the details!

tivadardanka.com/books/mathemat…
19
Read the unrolled thread here:

thepalindrome.substack.com/p/the-law-of-l…
20
If you have enjoyed this thread, share it with your friends, follow me, and subscribe to my newsletter!

Understanding mathematics is a superpower. I'll help you get it.

thepalindrome.substack.com
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement