Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference

4. The Greatest Theorem Never Told

4.1 Introduction

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we’ve been using this simple idea in every example thus far.

4.2 The Law of Large Numbers

Let Z_i be N independent samples from some probability distribution. According to the Law of Large Numbers, so long as the expected value E[Z] is not infinity, the following holds:

In words:

The average of a set of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will ...