缺失数据的极大似然估计：《Statistical Analysis with Missing Data》习题7.16

一、题目

a）极大似然估计

$X$为伯努利分布，并且 $\text{Pr}(X = 1) = 1 - \text{Pr}(X = 0) = \pi$，并且在给定 $X = j\ \ (j=0，1)$时， $Y$的分布为均值 $\mu_j$，方差 $\sigma^2$。

针对一份完整随机样本 $(x_i，y_i)，i=1，...，n$，计算 $(\pi，\mu_0，\mu_1，\sigma^2)$的极大似然估计并计算 $Y$的边际均值与方差。

b）缺失数据的极大似然估计

假设现在 $X$是完整的观测，但 $Y$有 $n-r$个值缺失，请使用第七章的方法，计算 $Y$的边际均值与方差。

c）从后验分布中生成参数

当先验分布表现为 $p(\pi，\mu_0，\mu_1，\text{log}\sigma^2) \propto \pi^{1/2}(1-\pi)^{1/2}$的形式，描述如何从参数为 $(\pi，\mu_0，\mu_1，\sigma^2)$的后验分布中抽出参数。

（注：前面的逗号均使用全角，后面公式中的逗号为半角，中文字中间的逗号为全角。）

二、解答

a）极大似然估计

写出联合密度函数，首先列出一个样本时的密度：
$\begin{aligned} & f(x_i,y_i|\mu_0,\mu_1,\sigma^2,\pi) \\ = & f(x_i | \pi) \cdot f(y_i|x_i, \mu_0,\mu_1,\sigma^2,\pi) \\ = & \pi^{x_i} \cdot (1 - \pi)^{1 - x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_1)^2}{2\sigma^2}\})^{x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_0)^2}{2\sigma^2}\})^{1 - x_i} \\ \end{aligned}$

$n$个样本的联合密度函数：
$\begin{aligned} & f(X,Y|\mu_0,\mu_0,\sigma^2,\pi) \\ = & \prod_{i = 1}^n f(x_i | \pi) \cdot f(y_i|x_i, \mu_0,\mu_1,\sigma^2,\pi) \\ = & \prod_{i = 1}^n \pi^{x_i} \cdot (1 - \pi)^{1 - x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_1)^2}{2\sigma^2}\})^{x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_0)^2}{2\sigma^2}\})^{1 - x_i} \\ \end{aligned}$

对数似然函数：
$\begin{aligned} & \text{ln} f(X,Y|\mu_0,\mu_1,\sigma^2,\pi) \\ = & \text{ln} \prod_{i = 1}^n f(x_i | \pi) \cdot f(y_i|x_i, \mu_0,\mu_1,\sigma^2,\pi) \\ = & \text{ln} \prod_{i = 1}^n \pi^{x_i} \cdot (1 - \pi)^{1 - x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_1)^2}{2\sigma^2}\})^{x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_0)^2}{2\sigma^2}\})^{1 - x_i} \\ = & \sum_{i = 1}^n \{x_i\text{ln} \pi + (1 - x_i) \text{ln} (1 - \pi) - \frac{x_i}{2} \text{ln} (2\pi\sigma^2) -\frac{x_i(y_i - \mu_1)^2}{2\sigma^2} - \frac{1-x_i}{2} \text{ln} (2\pi\sigma^2) -\frac{(1-x_i)(y_i - \mu_0)^2}{2\sigma^2} \} \\ = & \sum_{i = 1}^n x_i\text{ln} \pi + (n - \sum_{i = 1}^n x_i) \text{ln} (1 - \pi) - \frac{n}{2} \text{ln} (2\pi\sigma^2) - \sum_{i = 1}^n \frac{x_i(y_i - \mu_1)^2}{2\sigma^2} - \sum_{i = 1}^n \frac{(1-x_i)(y_i - \mu_0)^2}{2\sigma^2} \} \\ \end{aligned}$

对上式求偏导，使其等于 $0$即可得极大似然估计
$\begin{aligned} \frac{\partial \text{ln} f(X,Y|\mu_0,\mu_1,\sigma^2,\pi)}{\partial \mu_1} = & \frac{\partial \text{ln}f(X,Y|\mu_0,\mu_1,\sigma^2,\pi)}{\partial \mu_0} \\ = & \frac{\partial \text{ln} f(X,Y|\mu_0,\mu_1,\sigma^2,\pi)}{\partial \sigma^2} \\ = &\frac{\partial \text{ln} f(X,Y|\mu_0,\mu_1,\sigma^2,\pi)}{\partial \pi} \\ = & 0 \end{aligned}$

可解得：
$\begin{aligned} \hat{\pi} = & \frac{\sum_{i = 1}^n x_i}{n} \\ \hat{\mu_0} = & \frac{\sum_{i = 1}^n (1 - x_i) y_i}{\sum_{i = 1}^n (1 - x_i)}\\ \hat{\mu_1} = & \frac{\sum_{i = 1}^n x_i y_i}{\sum_{i = 1}^n x_i} \\ \hat{\sigma^2} = & \frac{\sum_{i = 1}^n y_i^2}{n} - \frac{[\sum_{i = 1}^n (1 - x_i) y_i]^2}{n\sum_{i = 1}^n (1 - x_i)} - \frac{(\sum_{i = 1}^n x_i y_i)^2}{n\sum_{i = 1}^n x_i} \\ \end{aligned}$

由于 $\hat{\sigma^2}$的计算相对麻烦，这里将其详细的计算过程写出：
$\begin{aligned} \frac{\partial \text{ln} f(X,Y|\mu_0,\mu_1,\sigma^2,\pi)}{\partial \sigma^2} = & 0 \\ -\frac{n}{2} \cdot -\frac{1}{2\pi \hat{\sigma^2}}\cdot 2\pi + \frac{ \sum_{i = 1}^n x_i(y_i - \hat{\mu_1})^2}{2(\hat{\sigma^2})^2} + \frac{\sum_{i = 1}^n (1-x_i)(y_i - \hat{\mu_0})^2}{2(\hat{\sigma^2})^2} \} = & 0 \\ \sum_{i = 1}^n x_i(y_i - \frac{\sum_{i = 1}^n x_i y_i}{\sum_{i = 1}^n x_i})^2 + \sum_{i = 1}^n (1-x_i)(y_i - \frac{\sum_{i = 1}^n (1 - x_i) y_i}{\sum_{i = 1}^n (1 - x_i)})^2 = & n\hat{\sigma^2} \\ \frac{\sum_{i = 1}^n y_i^2}{n} - \frac{[\sum_{i = 1}^n (1 - x_i) y_i]^2}{n\sum_{i = 1}^n (1 - x_i)} - \frac{(\sum_{i = 1}^n x_i y_i)^2}{n\sum_{i = 1}^n x_i} = & \hat{\sigma^2} \\ \end{aligned}$

均值与方差为：
将随机变量 $X$求和掉，可求得 $Y$的边际分布：
$Y = (1-\pi) Y_0 + \pi Y_1$
其中：
$\begin{aligned} Y_0 \sim & N(\mu_0, \sigma^2) \\ Y_1 \sim & N(\mu_1, \sigma^2) \\ \end{aligned}$

对其求期望与方差：
$EY = (1-\pi) \mu_0 + \pi \mu_1$
$Var(Y) = (1-\pi)^2 \sigma^2 + \pi^2 \sigma^2$
则 $Y$边际均值的估计为：
$(1-\hat{\pi}) \hat{\mu_0} + \hat{\pi} \hat{\mu_1}$
边际方差的估计为：
$(1-\hat{\pi})^2 \hat{\sigma^2} + \hat{\pi}^2 \hat{\sigma^2}$
将前面的估计得到的参数带入即可。

b）带缺失数据的极大似然估计

联合密度函数：
$\begin{aligned} & f(X,Y_{obs}|\mu_0,\mu_1,\sigma^2,\pi) \\ = & \prod_{i = 1}^r f(x_i, y_i | \mu_0,\mu_1,\sigma^2,\pi) \cdot \prod_{i = r+1}^n f(x_i|\pi) \\ = & \prod_{i = 1}^r f(x_i | \pi) f(y_i|x_i, \mu_0,\mu_1,\sigma^2,\pi) \cdot \prod_{i = r+1}^n f(x_i|\pi) \\ = & \prod_{i = 1}^n f(x_i|\pi) \cdot \prod_{i = 1}^r f(y_i|x_i, \mu_0,\mu_1,\sigma^2,\pi)\\ = & \prod_{i = 1}^n \pi^{x_i}(1 - \pi)^{1 - x_i} \cdot \prod_{i = 1}^r (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_1)^2}{2\sigma^2}\})^{x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_0)^2}{2\sigma^2}\})^{1 - x_i} \\ \end{aligned}$

同样对上式求对数与偏导，使其等于 $0$，可解得：
$\begin{aligned} \hat{\pi} = & \frac{\sum_{i = 1}^n x_i}{n} \\ \hat{\mu_1} = & \frac{\sum_{i = 1}^r x_i y_i}{\sum_{i = 1}^r x_i} \\ \hat{\mu_0} = & \frac{\sum_{i = 1}^r (1 - x_i) y_i}{\sum_{i = 1}^r (1 - x_i)}\\ \hat{\sigma^2} = & \frac{\sum_{i = 1}^r y_i^2}{r} - \frac{[\sum_{i = 1}^r (1 - x_i) y_i]^2}{r\sum_{i = 1}^r (1 - x_i)} - \frac{(\sum_{i = 1}^r x_i y_i)^2}{r\sum_{i = 1}^r x_i} \\ \end{aligned}$

同前面无缺失的情况， $Y$边际均值的估计为：
$(1-\hat{\pi}) \hat{\mu_0} + \hat{\pi} \hat{\mu_1}$
边际方差的估计为：
$(1-\hat{\pi})^2 \hat{\sigma^2} + \hat{\pi}^2 \hat{\sigma^2}$
同样将前面的带缺失数据的极大似然估计得到的参数带入即可。

c）从后验分布中生成参数

后验分布：
$\begin{aligned} & f(\mu_0,\mu_1,\sigma^2,\pi|X,Y_{obs}) \\ \propto & f(X,Y_{obs}|\mu_0,\mu_1,\sigma^2,\pi) \cdot f(\mu_0,\mu_1,\sigma^2,\pi) \\ \propto & \prod_{i = 1}^n \pi^{\frac{1}{2} + x_i}(1 - \pi)^{\frac{3}{2} - x_i} \cdot \prod_{i = 1}^r (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_1)^2}{2\sigma^2}\})^{x_i} \cdot (\frac{1}{\sqrt{2\pi\sigma^2}}\text{exp}\{-\frac{(y_i - \mu_0)^2}{2\sigma^2}\})^{1 - x_i} \\ \end{aligned}$

我们可以类似书上的141页进行参数任意函数 $g_d$的生成：

1. 从参数为 $(\frac{n}{2}+\sum_{i = i}^n x_i+1,\frac{3n}{2}-\sum_{i = i}^n x_i+1)$的Beta分布中抽取 $b_{t}$；从自由度为 $2n-2$的卡方分布中抽取 $x_{t}$；从标准正态分布中抽取相互独立的 $z_0$与 $z_1$；
2. 计算 $\phi^{(d)} = (\pi^{(d)}, \mu_1^{(d)}, \mu_0^{(d)}, {\sigma^2}^{(d)})$：（其中 $\hat{\sigma^2}, \hat{\mu_1}, \hat{\mu_0}$均为上一问所求）
   $\begin{aligned} \pi^{(d)} &= b_{t}\\ {\sigma^2}^{(d)} &= n \hat{\sigma^2} / x_t\\ \mu_1^{(d)} &= \hat{\mu_1} + z_0 ({\sigma^2}^{(d)} / \sum_{i = 1}^r x_i) ^ {1/2}\\ \mu_0^{(d)} &= \hat{\mu_0} + z_1 ({\sigma^2}^{(d)} / \sum_{i = 1}^r (1-x_i)) ^ {1/2}\\ \end{aligned}$