Chapter 5 (Limit Theorems): The Strong Law of Large Numbers (强大数定律)

本文为 $Introduction$ $to$ $Probability$ 的读书笔记

The Strong Law of Large Numbers

• The strong law of large numbers is similar to the weak law in that it also deals with the convergence of the sample mean to the true mean. It is different, however, because it refers to another type of convergence.
  • The weak law states that the probability $P(|M_n-\mu |\ge ϵ)$ of a significant deviation of $M_n$ from $\mu$ goes to zero as $n\to \infty$. Still, for any finite $n$, this probability can be positive and it is conceivable that once in a while, even if infrequently, $M_n$ deviates significantly from $\mu$. The weak law provides no conclusive information on the number of such deviations, but the strong law does.
  • According to the strong law, and with probability $1$, $M_n$ converges to $\mu$. This implies that for any given $ϵ>0$, the probability that the difference $|M_n-\mu |$ will exceed $ϵ$ an infinite number of times is equal to zero.

---

Example 5.13. Probabilities and Frequencies.

• Consider an event $A$ defined in terms of some probabilistic experiment. We consider a sequence of independent repetitions of the same experiment. and let $M_n$ be the fraction of the first $n$ repetitions in which $A$ occurs.
• The strong law of large numbers asserts that $M_n$ converges to $P(A)$, with probability 1.
• In contrast, the weak law of large numbers asserts that $M_n$ converges to $P(A)$ in probability (依概率收敛).
• We have often talked intuitively about the probability of an event $A$ as the frequency with which it occurs in an infinitely long sequence of independent trials. The strong law backs this intuition and establishes that the long-term frequency of occurrence of $A$ is indeed equal to $P(A)$, with essential certainty (the probability of this happening is 1).

---

Problem 18. The strong law of large numbers.
Let $X_1,X_2,...$ be a sequence of independent identically distributed random variables and assume that $E[X_i^4]<\infty$. Prove the strong law of large numbers.

SOLUTION

• We note that the assumption $E[X_i^4]<\infty$ implies that the expected value of the $X_i$ is finite. Indeed, using the inequality $|x|\le 1+x^4$, we have
  $$E[|X_i|]\le 1+E[X_i^4]<\infty$$
• Let us assume first that $E[X_i]=0$. We will show that
  $$E\left[\sum _{n=1}^{\infty }\frac{(X_1+...+X_n{)}^4}{n^4}\right]<\infty$$We have
  $$E\left[\sum _{n=1}^{\infty }\frac{(X_1+...+X_n{)}^4}{n^4}\right]=\frac{1}{n^4}\sum _{i_1=1}^n\sum _{i_2=1}^n\sum _{i_3=1}^n\sum _{i_4=1}^{n}E[X_{i1}{X}_{i2}{X}_{i3}{X}_{i4}]$$Let us consider the various terms in this sum. Since $E[X_i]=0$, if one of the indices is different from all of the other indices, the corresponding term is equal to zero. Therefore, the nonzero terms in the above sum are either of the form $E[X_i^4]$ (there are $n$ such terms), or of the form $E[X_i^2{X}_j^2]$, with $i\ne j$. Let us count how many terms there are of this form. Such terms are obtained in three different ways: by setting $i_1=i_2\ne i_3=i_4$, or by setting $i_1=i_3\ne i_2=i_4$ or by setting $i_1=i_4\ne i_2=i_3$. For each one of these three ways, we have $n$ choices for the first pair of indices, and $n-1$ choices for the second pair. We conclude that there are $3n(n-1)$ terms of this type. Thus,
  $$\begin{array}{rl}E[(X_1+...+X_n{)}^4]& =nE[X_1^4]+3n(n-1)E[X_1^2{X}_2^2]\end{array}$$Using the inequality $xy\le (x^2+y^2)/2$, we obtain $E[X_1^2{X}_2^2]\le E[X_1^4]$, and
  $$\begin{array}{rl}E[(X_1+...+X_n{)}^4]& \le (n+3n(n-1))E[X_1^4]\le 3n^{2}E[X_1^4]\end{array}$$It follows that
  $$\begin{array}{r}E\left[\sum _{n=1}^{\infty }\frac{(X_1+...+X_n{)}^4}{n^4}\right]=\sum _{n=1}^{\infty }\frac{1}{n^4}E[(X_1+...+X_n{)}^4]\le \sum _{n=1}^{\infty }\frac{3}{n^2}E[X_1^4]<\infty \end{array}$$where the last step uses the weill knoen property ${\sum }_{n=1}^{\infty }{n}^{-2}<\infty$. This impiles that $(X_1+...+X_n{)}^4/n^4$ converges to zero with probability 1, and therefore, $(X_1+\cdot \cdot \cdot +X_n)/n$ also converges to zero with probability 1, which is the st rong law of large numbers.
• For the more general case where the mean of the random variables $X_i$ is nonzero, the preceding argument establishes that $(X_1+\cdot \cdot \cdot +X_n-nE[X_1])/n$ converges to zero, which is the same as $(X_1+\cdot \cdot \cdot +X_n)/n$ converging to $E[X_1]$, with probability 1.

Convergence with Probability 1

• A proper interpretation of this type of convergence involves a sample space consisting of infinite sequences. It is best to think of the sample space as a set of infinite sequences $(y_1,y_2....)$ of real numbers: any such sequence is a possible outcome of the experiment. Let us now consider the set $A$ consisting of those sequences $(y_1,y_2...)$ whose long-term average is $c$. All of the probability is concentrated on this particular subset of the sample space. This does not mean that other sequences are impossible, only that they are extremely unlikely, in the sense that their total probability is zero.

---

Example 5.14.

• Let $X_1,X_2....$ be a sequence of independent random variables that are uniformly distributed in $[0,1]$, and let Y n = min ⁡ { X 1 , . . . . X n } Y_n= \min \{ X_1, .... X_n\} Yn​=min{ X1​,....Xn​}. We wish to show that $Y_n$ converges to 0, with probability 1.
• In any execution of the experiment, the sequence $Y_n$ is nonincreasing. Since this sequence is bounded below by zero, it must have a limit, which we denote by $Y$. Let us fix some $ϵ>0$,
  $$P(Y\ge ϵ)=P(X_1\ge ϵ...X_n\ge ϵ)=(1-ϵ{)}^n$$Since this is true for all $n$, we must have
  $$P(Y\ge ϵ)\le \underset{n\to \infty }{\lim }(1-ϵ{)}^n=0$$This shows that $P(Y\ge ϵ)=0$, for any positive $ϵ$. We conclude that $P(Y>0)=0$, which implies that $P(Y=0)=1$. Since $Y$ is the limit of $Y_n$, we see that $Y_n$ converges to zero with probability 1.

---

Problem 13.
Consider two sequences of random variables $X_1,X_2,...$ and $Y_1,Y_2,....$ Suppose that $X_n$ converges to $a$ and $Y_n$ converges to $b$, with probability 1. Show that $X_n+Y_n$ converges to $a+b$, with probability 1. Also, assuming that the random variables $Y_n$ cannot be equal to zero, show that $X_n/Y_n$ converges to $a/b$, with probability $1$.

SOLUTION

• Let $A$ (respectively, $B$) be the event that the sequence of values of the random variables $X_n$ (respectively, $Y_n$) does not converge to $a$ (respectively, $b$). Let $C$ be the event that the sequence of values of $X_n+Y_n$ does not converge to $a+b$ and notice that $C\subset A\cup B$. Hence,
  $$P(C)\le P(A\cup B)\le P(A)+P(B)=0$$Therefore, $P(C^C)=1$, or equivalently, $X_n+Y_n$ converges to $a+b$ with probability 1.
• For the convergence of $X_n/Y_n$, the argument is similar.

---

Problem 16.
Consider a sequence $Y_n$ of nonnegative random variables and suppose that
$$E\left[\sum _{n=1}^{\infty }{Y}_n\right]<\infty$$Show that $Y_n$ converges to 0, with probability 1.

• Note: This result provides a commonly used method for establishing convergence with probability 1.
• To evaluate the expectation of ${\sum }_{n=1}^{\infty }{Y}_n$, one typically uses the formula
  $$E\left[\sum _{n=1}^{\infty }{Y}_n\right]=\sum _{n=1}^{\infty }E[Y_n]$$The fact that the expectation and the infinite summation can be interchanged, for the case of nonnegative random variables, is known as the monotone convergence theorem (单调收敛定理), a fundamental result of probability theory, whose proof lies beyond the scope of this text.

SOLUTION

• We note that the infinite sum ${\sum }_{n=1}^{\infty }{Y}_n$ must be finite, with probability 1. Indeed, if it had a positive probability of being infinite, then its expectation would also be infinite. But if the sum of the values of the random variables $Y_n$ is finite, the sequence of these values must converge to zero. Since the probability of this event is equal to 1, it follows that the sequence $Y_n$ converges to zero, with probability 1.

---

Problem 17.
Consider a sequence of Bernoulli random variables $X_n$, and let $p_n=P(X_n=1)$ be the probability of success in the $n$th trial. Assuming that ${\sum }_{n=1}^{\infty }{p}_n<\infty$, show that the number of successes is finite, with probability 1. [Compare with Problem 48(b)]

SOLUTION

• Using the monotone convergence theorem (see above note), we have
  $$E\left[\sum _{n=1}^{\infty }{X}_n\right]=\sum _{n=1}^{\infty }E[X_n]=\sum _{n=1}^{\infty }{p}_n<\infty$$This implies that
  $$\sum _{n=1}^{\infty }{X}_n<\infty$$with probability 1.

“Convergence with probability 1” VS “Convergence in probability”

• Convergence with probability 1 implies convergence in probability, but the converse is not necessarily true.

Problem 15.
Suppose that a sequence $Y_1,Y_2,...$ of random variables converges to a real number $c$, with probability 1. Show that the sequence also converges to $c$ in probability.

SOLUTION

• Let $C$ be the event that the sequence of values of the random variables $Y_n$ converges to $c$. By assumption, we have $P(C)=1$. Fix some $ϵ>0$, and let $A_k$ be the event that $|Y_n-c|<ϵ$ for every $n\ge k$. If the sequence of values of the random variables $Y_n$ converges to $c$, then there must exist some $k$ such that for every $n\ge k$, this sequence of values is within less than $ϵ$ from $c$. Therefore, every element of $C$ belongs to $A_k$ for some $k$, or
  $$C\subset \bigcup _{k=1}^{\infty }{A}_k$$Note also that the sequence of events $A_k$ is monotonically increasing, in the sense that $A_k\subset A_{k+1}$ for all $k$. Finally, note that the event $A_k$ is a subset of the event { ∣ Y k − c ∣ < ϵ } \{|Y_k - c|< \epsilon\} { ∣Yk​−c∣<ϵ}. Therefore,
  $$\underset{k\to \infty }{\lim }P(|Y_k-c|<ϵ)\ge \underset{k\to \infty }{\lim }P(A_k)=P(\bigcup _{k=1}^{\infty }{A}_k)\ge P(C)=1$$It follows that
  $$\underset{k\to \infty }{\lim }P(|Y_k-c|\ge ϵ)=0$$

---

• Our last example illustrates the difference between convergence in probability and convergence with probability 1.

Example 5.15.

• Consider a discrete-time arrival process. The set of times is partitioned into consecutive intervals of the form I k = { 2 k , 2 k + 1 , . . . , 2 k + 1 − 1 } I_k = \{ 2^k, 2^{k}+1, ... , 2^{k+ 1} - 1\} Ik​={ 2k,2k+1,...,2k+1−1}. Note that the length of $I_k$ is $2^k$. During each interval $I_k$, there is exactly one arrival, and all times within an interval are equally likely. The arrival times within different intervals are assumed to be independent. Let us define $Y_n=1$ if there is an arrival at time $n$, and $Y_n=0$ if there is no arrival. We have $P(Y_n=1)=1/2^k$, if $n\in l_k$. Note that as $n$ increases, It belongs to intervals $I_k$ with increasingly large indices $k$. Consequently,
  $$\underset{n\to \infty }{\lim }P(Y_n\ne 0)=\underset{k\to \infty }{\lim }\frac{1}{2^k}=0$$and we conclude that $Y_n$ converges to 0 in probability.
• However, when we carry out the experiment, the total number of arrivals is infinite (one arrival during each interval $I_k$). Therefore, $Y_n$ is unity for infinitely many values of $n$, the event { lim ⁡ n → ∞ Y n = 0 } \{\lim_{n\rightarrow\infty}Y_n=0\} { limn→∞​Yn​=0} has zero probability, and we do not have convergence with prob­ability 1.
• Intuitively, the following is happening. At any given time, there is only a small (and diminishing with $n$) probability of a substantial deviation from 0, which implies convergence in probability. On the other hand. given enough time, a substantial deviation from 0 is certain to occur and for this reason, we do not have convergence with probability 1.