Steven R. Dunbar
Department of Mathematics
203 Avery Hall
University of Nebraska-Lincoln
Lincoln, NE 68588-0130
http://www.math.unl.edu
Voice: 402-472-3731
Fax: 402-472-8466

Topics in
Probability Theory and Stochastic Processes
Steven R. Dunbar

__________________________________________________________________________

The Weak Law of Large Numbers

_______________________________________________________________________

Rating

Rating

Mathematicians Only: prolonged scenes of intense rigor.

_______________________________________________________________________________________________

QuestionofDay

Question of the Day

Consider a fair (p = 12 = q) coin tossing game carried out for 1000 tosses. Explain in a sentence what the “law of averages” says about the outcomes of this game. Be as precise as possible.

_______________________________________________________________________________________________

Key Concepts

Key Concepts

  1. Markov’s Inequality: Let X be a random variable taking only non-negative values. Then for each a > 0
    X > a E Xa;

  2. Chebyshev’s Inequality: Let X be a random variable. Then for a > 0
    |X E X| a Var X a2

  3. Weak Law of Large Numbers: For ϵ > 0
    n Sn n p > ϵ 0  as n

    and the convergence is uniform in p.

  4. Let f be a real function that is defined and continuous on the interval [0, 1]. Then
    sup 0x1 f(x) k=0nf k n n kxk(1 x)nk 0

    as n .

__________________________________________________________________________

Vocabulary

Vocabulary

  1. The Weak Law of Large Numbers says that for ϵ > 0
    n Sn n p > ϵ 0  as n

    and the convergence is uniform in p.

  2. The polynomials
    Bn,k(t) = n kxk(1 x)nk

    are called the Bernstein polynomials.

__________________________________________________________________________

Mathematical Ideas

Mathematical Ideas

Proof of the Weak Law Using Chebyshev’s Inequality

Proposition 1 (Markov’s Inequality). Let X be a random variable taking only non-negative values. Then for each a > 0

X a E Xa.

Proof.

X a = E IXa =Xad x ad 1 aE X

Proposition 2 (Chebyshev’s Inequality). Let X be a random variable. Then for a > 0

|X E X| a Var X a2 .

Proof. This immediately follows from Markov’s inequality applied to the non-negative random variable (X E X)2. □

Theorem 3 (Weak Law of Large Numbers). For ϵ > 0

n Sn n p > ϵ 0  as n

and the convergence is uniform in p.

Remark. The notation n indicates that we are considering a family of probability measures on the sample space Ω. The Weak Law establishes the convergence of the sequence of measures in a particular way.

Proof. The variance of the random variable Sn is np(1 p). Rewrite the probability as the equivalent event:

n Sn n p > ϵ = n Sn pn > nϵ.

By Chebyshev’s inequality

n Sn np > nϵ Var Sn (nϵ)2 = p(1 p) ϵ2 1 n.

Since p(1 p) 14, the proof is complete. □

Remark. This inequality demonstrates that

n Sn n p > ϵ = O(1n)

uniformly in p.

Remark. Jacob Bernoulli originally proved the Weak Law of Large Numbers in 1713 for the special case when the Xi are binomial random variables. Bernoulli had to create an ingenious proof to establish the result, since Chebyshev’s inequality was not known at the time. The theorem then became known as Bernoulli’s Theorem. Simeon Poisson proved a generalization of Bernoulli’s binomial Weak Law and first called it the Law of Large Numbers. In 1929 the Russian mathematician Aleksandr Khinchin proved the general form of the Weak Law of Large Numbers presented here. Many other versions of the Weak Law are known, with hypotheses that do not require such stringent requirements as being identically distributed, and having finite variance.

Remark. Another proof of the Weak Law of Large Numbers using moment generating functions is in Mathematical Finance/Central Limit Theorem.

Bernstein’s Proof of the Weierstrass Approximation Theorem

Theorem 4. Let f be a real function defined and continuous on the interval [0, 1]. Then

sup 0x1 f(x) k=0nf k n n kxk(1 x)nk 0

as n .

Proof.

  1. Fix ϵ > 0. Since f continuous on the compact interval [0, 1] it is uniformly continuous on [0, 1]. Therefore there is an η > 0 such that |f(x) f(y)| < ϵ if |x y| < η.
  2. The expectation E f(Snn) can be expressed as a polynomial in p:
    E f Sn n = k=0nf k n n Sn = k = k=0nf k n n kpk(1 p)nk.

  3. By the Weak Law of Large Numbers, for the given ϵ > 0, there is an n0 such that
    n Sn n p > η < ϵ.

  4. E f Sn n f(p) = k=0n f k n f(p) n Sn = k.

  5. Apply the triangle inequality to the right hand side and express in terms of two summations: |k np|η f k n f(p) n Sn = k + |k np|>η |f k n| + |f(p)| n Sn = k

    Note the second application of the triangle inequality on the second summation.

  6. Now estimate the terms:
    |k np|ηϵn Sn = k + |k np|>η2 sup 0x1|f(x)|n Sn = k

  7. Finally, do the addition over the individual values of the probabilities over single values to re-write them as probabilities over events:
    = ϵn Sn n p η + 2 sup 0x1|f(x)|n Sn n p > η

  8. Now apply the Weak Law to the second term to see that:
    E f Sn n f(p) < ϵ + 2ϵ sup 0x1|f(x)|.

    This shows that E f Sn n f(p) can be made arbitrarily small, uniformly with respect to p, by picking n sufficiently large.

Remark. The polynomials

Bn,k(t) = n kxk(1 x)nk

are called the Bernstein polynomials. The Bernstein polynomials have several useful properties:

  1. Bi,n(t) = Bni,n(1 t)
  2. Bi,n(t) 0
  3. i=0nB i,n(t) = 1 for 0 t 1.

Corollary 1. A polynomial of degree n uniformly approximating the continuous function f(x) on the interval [a,b] is

k=0nf a + (b a)k n n k x a b ak b x b ank

Sources

This section is adapted from: Heads or Tails, by Emmanuel Lesigne, Student Mathematical Library Volume 28, American Mathematical Society, Providence, 2005, Chapter 5, [3].

_______________________________________________________________________________________________

Problems to Work

Problems to Work for Understanding

  1. Suppose X is a continuous random variable with mean and variance both equal to 20. What can be said about 0 X 40?
  2. Suppose X is an exponentially distributed random variable with mean E X = 1. For x = 0.5, 1 , and 2 , compare X x with the Markov inequality bound.
  3. Suppose X is a Bernoulli random variable with X = 1 = p and X = 0 = 1 p = q. Compare X 1 with the Markov inequality bound.
  4. Let X1,X2,,X10 be independent Poisson random variables with mean 1. First use the Markov Inequality to get a bound on X1 + + X10 > 15. Next find the exact probability that X1 + + X10 > 15 using that the fact that the sum of independent Poisson random variables with parameters λ1, λ2 is again Poisson with parameter λ1 + λ2.
  5. Consider a fair (p = 12 = q) coin tossing game carried out for n = 100 tosses. Calculate the exact value
    n Sn n p > 110

    and compare it to the estimates in the proof of the Weak Law of Large Numbers.

  6. Calculate the Bernstein polynomial approximation of sin(πx) of degree 1, 2, and 3 and plot the graphs of sin(πx) and the approximations.
  7. Calculate the Bernstein polynomial approximation of cos(πx) of degree 1, 2, and 3 and plot the graphs of cos(πx) and the approximations.
  8. Calculate the Bernstein polynomial approximation of exp(πx) of degree 1, 2, and 3 and plot the graphs of exp(πx) and the approximations.

__________________________________________________________________________

Links

Reading Suggestion:

References

[1]   Leo Breiman. Probability. SIAM, 1992.

[2]   William Feller. An Introduction to Probability Theory and Its Applications, Volume I, volume I. John Wiley and Sons, third edition, 1973. QA 273 F3712.

[3]   Emmanuel Lesigne. Heads or Tails: An Introduction to Limit Theorems in Probability, volume 28 of Student Mathematical Library. American Mathematical Society, 2005.

__________________________________________________________________________

Links

Outside Readings and Links:

  1. Virtual Laboratories in Probability and Statistics / Binomial.
  2. Weisstein, Eric W. “Weak Law of Large Numbers.” From MathWorld–A Wolfram Web Resource. Weak Law of Large Numbers..
  3. Wikipedia, Weak Law of Large Numbers.

__________________________________________________________________________

I check all the information on each page for correctness and typographical errors. Nevertheless, some errors may occur and I would be grateful if you would alert me to such errors. I make every reasonable effort to present current and accurate information for public use, however I do not guarantee the accuracy or timeliness of information on this website. Your use of the information from this website is strictly voluntary and at your risk.

I have checked the links to external sites for usefulness. Links to external websites are provided as a convenience. I do not endorse, control, monitor, or guarantee the information contained in any external website. I don’t guarantee that the links are active at all times. Use the links here with the same caution as you would all information on the Internet. This website reflects the thoughts, interests and opinions of its author. They do not explicitly represent official positions or policies of my employer.

Information on this website is subject to change without notice.

Steve Dunbar’s Home Page, http://www.math.unl.edu/~sdunbar1

Email to Steve Dunbar, sdunbar1 at unl dot edu

Last modified: Processed from LATEX source on May 25, 2011