Steven R. Dunbar
Department of Mathematics
203 Avery Hall
Lincoln, NE 68588-0130
http://www.math.unl.edu
Voice: 402-472-3731
Fax: 402-472-8466

Topics in
Probability Theory and Stochastic Processes
Steven R. Dunbar

__________________________________________________________________________

The Weak Law of Large Numbers

_______________________________________________________________________ ### Rating

Mathematicians Only: prolonged scenes of intense rigor.

_______________________________________________________________________________________________ ### Question of the Day

Consider a fair ($p=1∕2=q$) coin tossing game carried out for 1000 tosses. Explain in a sentence what the “law of averages” says about the outcomes of this game. Be as precise as possible.

_______________________________________________________________________________________________ ### Key Concepts

1. Markov’s Inequality: Let $X$ be a random variable taking only non-negative values. Then for each $a>0$
$ℙ\left[X>a\right]\le E\left[X\right]∕a;$

2. Chebyshev’s Inequality: Let $X$ be a random variable. Then for $a>0$
$ℙ\left[|X-E\left[X\right]|\ge a\right]\le \frac{Var\left[X\right]}{{a}^{2}}$

3. Weak Law of Large Numbers: For $ϵ>0$

and the convergence is uniform in $p$.

4. Let $f$ be a real function that is defined and continuous on the interval $\left[0,1\right]$. Then
$\underset{0\le x\le 1}{sup}\left|f\left(x\right)-\sum _{k=0}^{n}f\left(\frac{k}{n}\right)\left(\genfrac{}{}{0.0pt}{}{n}{k}\right){x}^{k}{\left(1-x\right)}^{n-k}\right|\to 0$

as $n\to \infty$.

__________________________________________________________________________ ### Vocabulary

1. The Weak Law of Large Numbers says that for $ϵ>0$

and the convergence is uniform in $p$.

2. The polynomials
${B}_{n,k}\left(t\right)=\left(\genfrac{}{}{0.0pt}{}{n}{k}\right){x}^{k}{\left(1-x\right)}^{n-k}$

are called the Bernstein polynomials.

__________________________________________________________________________ ### Mathematical Ideas

#### Proof of the Weak Law Using Chebyshev’s Inequality

Proposition 1 (Markov’s Inequality). Let $X$ be a random variable taking only non-negative values. Then for each $a>0$

$ℙ\left[X\ge a\right]\le E\left[X\right]∕a.$

Proof.

$\begin{array}{llll}\hfill ℙ\left[X\ge a\right]& =E\left[{I}_{X\ge a}\right]\phantom{\rule{2em}{0ex}}& \hfill & \phantom{\rule{2em}{0ex}}\\ \hfill & ={\int }_{X\ge a}\phantom{\rule{3.26288pt}{0ex}}dℙ\left[\right]\phantom{\rule{2em}{0ex}}& \hfill & \phantom{\rule{2em}{0ex}}\\ \hfill & \le \int \frac{x}{a}\phantom{\rule{3.26288pt}{0ex}}dℙ\left[\right]\phantom{\rule{2em}{0ex}}& \hfill & \phantom{\rule{2em}{0ex}}\\ \hfill & \le \frac{1}{a}E\left[X\right]\phantom{\rule{2em}{0ex}}& \hfill & \phantom{\rule{2em}{0ex}}\end{array}$

Proposition 2 (Chebyshev’s Inequality). Let $X$ be a random variable. Then for $a>0$

$ℙ\left[|X-E\left[X\right]|\ge a\right]\le \frac{Var\left[X\right]}{{a}^{2}}.$

Proof. This immediately follows from Markov’s inequality applied to the non-negative random variable ${\left(X-E\left[X\right]\right)}^{2}$. □

Theorem 3 (Weak Law of Large Numbers). For $ϵ>0$

and the convergence is uniform in $p$.

Remark. The notation $ℙn\left[\right]$ indicates that we are considering a family of probability measures on the sample space $\Omega$. The Weak Law establishes the convergence of the sequence of measures in a particular way.

Proof. The variance of the random variable ${S}_{n}$ is $np\left(1-p\right)$. Rewrite the probability as the equivalent event:

$ℙn\left[\left|\frac{{S}_{n}}{n}-p\right|>ϵ\right]=ℙn\left[\left|{S}_{n}-pn\right|>nϵ\right].$

By Chebyshev’s inequality

$ℙn\left[\left|{S}_{n}-np\right|>nϵ\right]\le \frac{Var\left[{S}_{n}\right]}{{\left(nϵ\right)}^{2}}=\frac{p\left(1-p\right)}{{ϵ}^{2}}\frac{1}{n}.$

Since $p\left(1-p\right)\le 1∕4$, the proof is complete. □

Remark. This inequality demonstrates that

$ℙn\left[\left|\frac{{S}_{n}}{n}-p\right|>ϵ\right]=O\left(1∕n\right)$

uniformly in $p$.

Remark. Jacob Bernoulli originally proved the Weak Law of Large Numbers in 1713 for the special case when the ${X}_{i}$ are binomial random variables. Bernoulli had to create an ingenious proof to establish the result, since Chebyshev’s inequality was not known at the time. The theorem then became known as Bernoulli’s Theorem. Simeon Poisson proved a generalization of Bernoulli’s binomial Weak Law and first called it the Law of Large Numbers. In 1929 the Russian mathematician Aleksandr Khinchin proved the general form of the Weak Law of Large Numbers presented here. Many other versions of the Weak Law are known, with hypotheses that do not require such stringent requirements as being identically distributed, and having finite variance.

Remark. Another proof of the Weak Law of Large Numbers using moment generating functions is in Mathematical Finance/Central Limit Theorem.

#### Bernstein’s Proof of the Weierstrass Approximation Theorem

Theorem 4. Let $f$ be a real function defined and continuous on the interval $\left[0,1\right]$. Then

$\underset{0\le x\le 1}{sup}\left|f\left(x\right)-\sum _{k=0}^{n}f\left(\frac{k}{n}\right)\cdot \left(\genfrac{}{}{0.0pt}{}{n}{k}\right){x}^{k}{\left(1-x\right)}^{n-k}\right|\to 0$

as $n\to \infty$.

Proof.

1. Fix $ϵ>0$. Since $f$ continuous on the compact interval $\left[0,1\right]$ it is uniformly continuous on $\left[0,1\right]$. Therefore there is an $\eta >0$ such that $|f\left(x\right)-f\left(y\right)|<ϵ$ if $|x-y|<\eta$.
2. The expectation $E\left[f\left({S}_{n}∕n\right)\right]$ can be expressed as a polynomial in $p$:
$E\left[f\left(\frac{{S}_{n}}{n}\right)\right]=\sum _{k=0}^{n}f\left(\frac{k}{n}\right)\cdot ℙn\left[{S}_{n}=k\right]=\sum _{k=0}^{n}f\left(\frac{k}{n}\right)\cdot \left(\genfrac{}{}{0.0pt}{}{n}{k}\right){p}^{k}{\left(1-p\right)}^{n-k}.$

3. By the Weak Law of Large Numbers, for the given $ϵ>0$, there is an ${n}_{0}$ such that
$ℙn\left[\left|\frac{{S}_{n}}{n}-p\right|>\eta \right]<ϵ.$

4. $\left|E\left[f\left(\frac{{S}_{n}}{n}\right)-f\left(p\right)\right]\right|=\left|\sum _{k=0}^{n}\left(f\left(\frac{k}{n}\right)-f\left(p\right)\right)ℙn\left[{S}_{n}=k\right]\right|.$

5. Apply the triangle inequality to the right hand side and express in terms of two summations: $\begin{array}{c}\le \sum _{|\frac{k}{n}-p|\le \eta }\left(f\left(\frac{k}{n}\right)-f\left(p\right)\right)ℙn\left[{S}_{n}=k\right]+\\ \sum _{|\frac{k}{n}-p|>\eta }\left(|f\left(\frac{k}{n}\right)|+|f\left(p\right)|\right)ℙn\left[{S}_{n}=k\right]\end{array}$

Note the second application of the triangle inequality on the second summation.

6. Now estimate the terms:
$\le \sum _{|\frac{k}{n}-p|\le \eta }ϵℙn\left[{S}_{n}=k\right]+\sum _{|\frac{k}{n}-p|>\eta }2\underset{0\le x\le 1}{sup}|f\left(x\right)|ℙn\left[{S}_{n}=k\right]$

7. Finally, do the addition over the individual values of the probabilities over single values to re-write them as probabilities over events:
$=ϵℙn\left[\left|\frac{{S}_{n}}{n}-p\right|\le \eta \right]+2\underset{0\le x\le 1}{sup}|f\left(x\right)|ℙn\left[\left|\frac{{S}_{n}}{n}-p\right|>\eta \right]$

8. Now apply the Weak Law to the second term to see that:
$\left|E\left[f\left(\frac{{S}_{n}}{n}\right)-f\left(p\right)\right]\right|<ϵ+2ϵ\underset{0\le x\le 1}{sup}|f\left(x\right)|.$

This shows that $\left|E\left[f\left(\frac{{S}_{n}}{n}\right)-f\left(p\right)\right]\right|$ can be made arbitrarily small, uniformly with respect to $p$, by picking $n$ sufficiently large.

Remark. The polynomials

${B}_{n,k}\left(t\right)=\left(\genfrac{}{}{0.0pt}{}{n}{k}\right){x}^{k}{\left(1-x\right)}^{n-k}$

are called the Bernstein polynomials. The Bernstein polynomials have several useful properties:

1. ${B}_{i,n}\left(t\right)={B}_{n-i,n}\left(1-t\right)$
2. ${B}_{i,n}\left(t\right)\ge 0$
3. ${\sum }_{i=0}^{n}{B}_{i,n}\left(t\right)=1$ for $0\le t\le 1$.

Corollary 1. A polynomial of degree $n$ uniformly approximating the continuous function $f\left(x\right)$ on the interval $\left[a,b\right]$ is

$\sum _{k=0}^{n}f\left(a+\left(b-a\right)\frac{k}{n}\right)\left(\genfrac{}{}{0.0pt}{}{n}{k}\right){\left(\frac{x-a}{b-a}\right)}^{k}{\left(\frac{b-x}{b-a}\right)}^{n-k}$

#### Sources

This section is adapted from: Heads or Tails, by Emmanuel Lesigne, Student Mathematical Library Volume 28, American Mathematical Society, Providence, 2005, Chapter 5, .

_______________________________________________________________________________________________ ### Problems to Work for Understanding

1. Suppose $X$ is a continuous random variable with mean and variance both equal to $20$. What can be said about $ℙ\left[0\le X\le 40\right]$?
2. Suppose X is an exponentially distributed random variable with mean $E\left[X\right]=1$. For $x=0.5$, $1$ , and $2$ , compare $ℙ\left[X\ge x\right]$ with the Markov inequality bound.
3. Suppose $X$ is a Bernoulli random variable with $ℙ\left[X=1\right]=p$ and $ℙ\left[X=0\right]=1-p=q$. Compare $ℙ\left[X\ge 1\right]$ with the Markov inequality bound.
4. Let ${X}_{1},{X}_{2},\dots ,{X}_{10}$ be independent Poisson random variables with mean $1$. First use the Markov Inequality to get a bound on $ℙ\left[{X}_{1}+\cdots +{X}_{10}>15\right]$. Next find the exact probability that $ℙ\left[{X}_{1}+\cdots +{X}_{10}>15\right]$ using that the fact that the sum of independent Poisson random variables with parameters ${\lambda }_{1}$, ${\lambda }_{2}$ is again Poisson with parameter ${\lambda }_{1}+{\lambda }_{2}$.
5. Consider a fair ($p=1∕2=q$) coin tossing game carried out for $n=100$ tosses. Calculate the exact value
$ℙn\left[\left|\frac{{S}_{n}}{n}-p\right|>1∕10\right]$

and compare it to the estimates in the proof of the Weak Law of Large Numbers.

6. Calculate the Bernstein polynomial approximation of $sin\left(\pi x\right)$ of degree $1$, $2$, and $3$ and plot the graphs of $sin\left(\pi x\right)$ and the approximations.
7. Calculate the Bernstein polynomial approximation of $cos\left(\pi x\right)$ of degree $1$, $2$, and $3$ and plot the graphs of $cos\left(\pi x\right)$ and the approximations.
8. Calculate the Bernstein polynomial approximation of $exp\left(\pi x\right)$ of degree $1$, $2$, and $3$ and plot the graphs of $exp\left(\pi x\right)$ and the approximations.

__________________________________________________________________________ ### References

   Leo Breiman. Probability. SIAM, 1992.

   William Feller. An Introduction to Probability Theory and Its Applications, Volume I, volume I. John Wiley and Sons, third edition, 1973. QA 273 F3712.

   Emmanuel Lesigne. Heads or Tails: An Introduction to Limit Theorems in Probability, volume 28 of Student Mathematical Library. American Mathematical Society, 2005.

__________________________________________________________________________ 1. Virtual Laboratories in Probability and Statistics / Binomial.
2. Weisstein, Eric W. “Weak Law of Large Numbers.” From MathWorld–A Wolfram Web Resource. Weak Law of Large Numbers..
3. Wikipedia, Weak Law of Large Numbers.

__________________________________________________________________________

I check all the information on each page for correctness and typographical errors. Nevertheless, some errors may occur and I would be grateful if you would alert me to such errors. I make every reasonable effort to present current and accurate information for public use, however I do not guarantee the accuracy or timeliness of information on this website. Your use of the information from this website is strictly voluntary and at your risk.

I have checked the links to external sites for usefulness. Links to external websites are provided as a convenience. I do not endorse, control, monitor, or guarantee the information contained in any external website. I don’t guarantee that the links are active at all times. Use the links here with the same caution as you would all information on the Internet. This website reflects the thoughts, interests and opinions of its author. They do not explicitly represent official positions or policies of my employer.

Information on this website is subject to change without notice.