Chapter 4 (Further Topics on Random Variables): Covariance and Correlation (协方差和相关)

本文为 $Introduction$ $to$ $Probability$ 的读书笔记

Covariance

• The covariance of two random variables $X$ and $Y$, denoted by $cov(X,Y)$, is defined by
  $$\begin{array}{rl}cov(X,Y)& =E[(X-E[X])(Y-E[Y])]\\ & =E[XY]-E[X]E[Y]\end{array}$$
• When $cov(X,Y)=0$, we say that $X$ and $Y$ are uncorrelated. Roughly speaking, a positive or negative covariance indicates that the values of $X-E[X]$ and $Y-E[Y]$ obtained in a single experiment “tend” to have the same or the opposite sign, respectively. Thus the sign of the covariance provides an important qualitative indicator of the relationship between $X$ and $Y$.
  

---

• We record a few properties of covariances that are easily derived from the definition: for any random variables $X$, $Y$, and $Z$, and any scalars $a$ and $b$, we have
  $$\begin{gathered} cov(X,X)=var(X) \\ cov(X,aY+b)=a\cdot cov(X,Y) \\ cov(X,Y+Z)=cov(X,Y)+cov(X,Z) \end{gathered}$$
• Assume that $X$ and $Y$ satisfy
  $$E[X|Y=y]=E[X],forally$$Then, assuming $X$ and $Y$ are discrete, the total expectation theorem implies that
  $$\begin{array}{rl}E[XY]& =\sum _y{p}_Y(y)E[XY|Y=y]=\sum _{y}y{p}_Y(y)E[X|Y=y]\\ & =\sum _{y}y{p}_Y(y)E[X]=E[X]E[Y]\end{array}$$so $X$ and $Y$ are uncorrelated. The argument for the continuous case is similar.
• Note thate if $X$ and $Y$ are independent, we have $cov(X,Y)=E[XY]-E[X]E[Y]=0$. Thus, if $X$ and $Y$ are independent, they are also uncorrelated. However, the converse is generally not true, as illustrated by the following example.

Example 4.13.

• The pair of random variables $(X,Y)$ takes the values $(1,0),(0,1),(-1,0)$ and $(0,-1)$. each with probability $1/4$. Therefore,
  $$cov(X,Y)=E[XY]-E[X]E[Y]=0-0=0$$and $X$ and $Y$ are uncorrelated. However, $X$ and $Y$ are not independent since, for example, a nonzero value of $X$ fixes the value of $Y$ to zero.

---

• The correlation coefficient (相关系数) $\rho (X,Y)$ of two random variables $X$ and $Y$ that have nonzero variances is defined as
  $$\rho (X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}$$(The simpler notation $\rho$ will also be used when $X$ and $Y$ are clear from the context.)
• It may be viewed as a normalized version of the covariance $cov(X,Y)$, and infact, it can be shown that $\rho$ ranges from $-1$ to $1$.
• If $\rho >0$ (or $\rho <0$), then the values of $X-E[X]$ and $Y-E[Y]$ “tend” to have the same (or opposite, respectively) sign. The size of $|\rho |$ normalized measure of the extent to which this is true. In fact, always assuming that $X$ and $Y$ have positive variances, it can be shown that $\rho =1$ (or $\rho =-1$) if and only if there exists a positive (or negative, respectively) constant $c$ such that
  $$Y-E[Y]=c(X-E[X])$$

---

Problem 20. Schwarz inequality 施瓦兹不等式
Show that for any random variables $X$ and $Y$, we have
$$(E[XY]{)}^2\le E[X^2]E[Y^2]$$

SOLUTION

• We may assume that $E[Y^2]\ne 0$; otherwise, we have $Y=0$ with probability 1, and hence $E[XY]=0$. so the inequality holds. We have
  $$\begin{array}{rl}0& \le E[(X-\frac{E[XY]}{E[Y^2]}Y{)}^2]\\ & =E[X^2-2\frac{E[XY]}{E[Y^2]}XY+\frac{(E[XY]{)}^2}{(E[Y^2]{)}^2}{Y}^2]\\ & =E[X^2]-2\frac{E[XY]}{E[Y^2]}E[XY]+\frac{(E[XY]{)}^2}{(E[Y^2]{)}^2}E[Y^2]\\ & =E[X^2]-\frac{(E[XY]{)}^2}{E[Y^2]}\end{array}$$i.e., $(E[XY]{)}^2\le E[X^2]E[Y^2]$

---

Problem 21. Correlation coefficient.
Consider the correlation coefficient
$$\rho (X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}$$of two random variables $X$ and $Y$ that have positive variances. Show that:

• $(a)$ $|\rho (X,Y)|\le 1$.
  [Hint: Use the Schwarz inequality from the preceding problem.]
• $(b)$ If $Y-E[Y]$ is a positive (or negative) multiple of $X-E[X]$, then $\rho (X,Y)=1$ [or $\rho (X.Y)=-1$, respectively].
• $(c)$ If $\rho (X,Y)=1$ [or $\rho (X,Y)=-1$], then, with probability 1, $Y-E[Y]$ is a positive (or negative. respectively) multiple of $X-E[X]$.

SOLUTION

• $(a)$ Let $\tilde{X}=X-E[X]$ and $\tilde{Y}=Y-E[Y]$. Using the Schwarz inequality, we get
  $$\rho (X,Y{)}^2=\frac{(E[\tilde{X}\tilde{Y}]{)}^2}{E[{\tilde{X}}^2]E[{\tilde{Y}}^2]}\le 1$$and hence $|\rho (X,Y)|\le 1$.
• $(b)$ If $\tilde{Y}=a\tilde{X}$, then
  $$\rho (X,Y)=\frac{E[\tilde{X}a\tilde{X}]}{\sqrt{E[\widetilde{X^2}]E[(a\tilde{X}{)}^2]}}=\frac{a}{|a|}$$
• $(c)$ If $|\rho (X,Y)|=1$, the calculation in the solution of Problem 20 yields
  $$\begin{array}{rl}E[(\tilde{X}-\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}{)}^2]& =E[{\tilde{X}}^2]-\frac{(E[\tilde{X}\tilde{Y}]{)}^2}{E[{\tilde{Y}}^2]}\\ & =E[{\tilde{X}}^2](1-(\rho (X,Y){)}^2)\\ & =0\end{array}$$Thus, with probability 1, the random variable
  $$\tilde{X}-\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}$$is equal to zero. It follows that, with probability 1,
  $$\tilde{X}=\frac{E[\tilde{X}\tilde{Y}]}{E[{\tilde{Y}}^2]}\tilde{Y}=\sqrt{\frac{E[{\tilde{X}}^2]}{E[{\tilde{Y}}^2]}}\rho (X,Y)\tilde{Y}$$i.e., the sign of the constant ratio of $X$ and $Y$ is determined by the sign of $\rho (X,Y)$.

Variance of the Sum of Random Variables

• If $X_1,X_2,...,X_n$ are random variables with finite variance, we have
  $$var(X_1+X_2)=var(X_1)+var(X_2)+2cov(X_1,X_2)$$and, more generally,
  v a r ( ∑ i = 1 n X i ) = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) var(\sum_{i=1}^nX_i)=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) var(i=1∑n​Xi​)=i=1∑n​var(Xi​)+{ (i,j)∣i​=j}∑​cov(Xi​,Xj​)

PROOF

• For brevity, we denote ${\tilde{X}}_i=X_i-E[X_i]$, then
  v a r ( ∑ i = 1 n X i ) = E [ ( ∑ i = 1 n X ~ i ) 2 ] = E [ ∑ i = 1 n ∑ j = 1 n X ~ i X ~ j ] = ∑ i = 1 n ∑ j = 1 n E [ X ~ i X ~ j ] = ∑ i = 1 n E [ X ~ i 2 ] + ∑ { ( i , j ) ∣ i ≠ j } E [ X ~ i X ~ j ] = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) \begin{aligned}var(\sum_{i=1}^nX_i)&=E[(\sum_{i=1}^n\tilde X_i)^2] \\&=E[\sum_{i=1}^n\sum_{j=1}^n\tilde X_i\tilde X_j] \\&=\sum_{i=1}^n\sum_{j=1}^nE[\tilde X_i\tilde X_j] \\&=\sum_{i=1}^nE[\tilde X_i^2]+\sum_{\{(i,j)|i\neq j\}}E[\tilde X_i\tilde X_j] \\&=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) \end{aligned} var(i=1∑n​Xi​)​=E[(i=1∑n​X~i​)2]=E[i=1∑n​j=1∑n​X~i​X~j​]=i=1∑n​j=1∑n​E[X~i​X~j​]=i=1∑n​E[X~i2​]+{ (i,j)∣i​=j}∑​E[X~i​X~j​]=i=1∑n​var(Xi​)+{ (i,j)∣i​=j}∑​cov(Xi​,Xj​)​

---

Example 4.15.
$n$ people throw their hats in a box and then pick a hat at random. Let us find the variance of $X$, the number of people who pick their own hat.

SOLUTION

• We have
  $$X=X_1+\cdot \cdot \cdot +X_n$$where $X_1$ is the random variable that takes the value $1$ if the $i$th person selects his/her own hat, and takes the value $0$ otherwise. Noting that $X_i$ is Bernoulli with parameter $p=P(X_i=1)=1/n$, we obtain
  $$\begin{array}{rl}E[X_i]& =\frac{1}{n}\\ var(X_i)& =\frac{1}{n}(1-\frac{1}{n})\end{array}$$For $i\ne j$, we have
  $$\begin{array}{rl}cov(X_i,X_j)& =E[X_i{X}_j]-E[X_i]E[X_j]\\ & =P(X_i=1and{X}_j=1)-\frac{1}{n^2}\\ & =P(X_i=1)P(X_j=1|X_i=1)-\frac{1}{n^2}\\ & =\frac{1}{n}\cdot \frac{1}{n-1}-\frac{1}{n^2}\\ & =\frac{1}{n^2(n-1)}\end{array}$$Therefore,
  v a r ( X ) = v a r ( ∑ i = 1 n X i ) = ∑ i = 1 n v a r ( X i ) + ∑ { ( i , j ) ∣ i ≠ j } c o v ( X i , X j ) = n ⋅ 1 n ( 1 − 1 n ) + n ( n − 1 ) ⋅ 1 n 2 ( n − 1 ) = 1 \begin{aligned}var(X)&=var(\sum_{i=1}^nX_i) \\&=\sum_{i=1}^nvar(X_i)+\sum_{\{(i,j)|i\neq j\}}cov(X_i,X_j) \\&=n\cdot \frac{1}{n}(1-\frac{1}{n})+n(n-1)\cdot \frac{1}{n^2(n-1)} \\&=1 \end{aligned} var(X)​=var(i=1∑n​Xi​)=i=1∑n​var(Xi​)+{ (i,j)∣i​=j}∑​cov(Xi​,Xj​)=n⋅n1​(1−n1​)+n(n−1)⋅n2(n−1)1​=1​