[Machine Learning] Expectation Maximization Algorithm (EM algorithm) analysis: Expectation Maximization Algorithm

[Machine Learning] Expectation Maximization Algorithm (EM algorithm): Expectation Maximization Algorithm

1 Introduction

EM algorithm, the full name is Expectation Maximization Algorithm. The expectation maximization algorithm is an iterative algorithm used for maximum likelihood estimation or maximum posterior probability estimation of probabilistic parameter models containing hidden variables (Hidden Variable).

We often find the model parameters of the sample from the sample observation data. The most common method is to maximize the log-likelihood function of the model distribution.

• However, in some cases, the observed data we obtain contains unobserved implicit data. At this time, we do not know the implicit data and model parameters, so we cannot directly use the maximum log-likelihood function to obtain the parameters of the model distribution. .

How to do it? This is where the EM algorithm comes in handy.

• The idea of ​​the EM algorithm to solve this problem is to use a heuristic iterative method. Since we cannot directly obtain the model distribution parameters, we can first guess the implicit data (step E of the EM algorithm), and then based on the observed data and guessed implicit data Let's maximize the log-likelihood and solve for our model parameters (the M step of the EM algorithm).
• Since our previous hidden data was guessed, the model parameters obtained at this time are generally not the results we want. But it doesn’t matter. Based on the currently obtained model parameters, we continue to guess the hidden data (the E step of the EM algorithm), and then continue to maximize the log likelihood and solve our model parameters (the M step of the EM algorithm). By analogy, iteration continues until the model distribution parameters basically remain unchanged, the algorithm converges, and appropriate model parameters are found.

As can be seen from the above description, the EM algorithm is an algorithm for iteratively solving the maximum value. At the same time, the algorithm is divided into two steps in each iteration, the E step and the M step. Iteratively update the hidden data and model distribution parameters round by round until convergence, that is, the model parameters we need are obtained.

• One of the most intuitive ways to understand the idea of ​​EM algorithm is the K-Means algorithm. See the principle of K-Means clustering algorithm written before. In K-Means clustering, the centroid of each cluster is the hidden data.
• We will assume K initial centroids, which is the E step of the EM algorithm; then calculate the nearest centroid of each sample, and cluster the samples to the nearest centroid, which is the M step of the EM algorithm. Repeat this E-step and M-step until the center of mass no longer changes, thus completing K-Means clustering.

Of course, the K-Means algorithm is relatively simple, but problems in practice are often not that simple. Next we need to describe it accurately in mathematical language.

2. Mathematical description of EM algorithm

For $M$ sample dataX = { x 1 , x 2 , x 3 , . . . x M } X = \{ x_1, x_2, x_3,... \ x_M \}X={ x1​,x2​,x3​,... xM​} , find the model parameters$\theta$ , the log-likelihood function of the maximizing model distribution is as follows:
$$\begin{array}{r}{i}^{\ast }=\underset{i}{\mathrm{argmax}}\sum _{i=1}^M\log P_{}(x_i)\end{array}$$

Assume that the observed data we obtain contains unobserved implicit data: C = { c 1 , c 2 , c 3 , . . . z K } C = \{ c_1, c_2, c_3,... \ z_K \}C={ c1​,c2​,c3​,... zK​}                                                                                      ⁡ max
$$\begin{array}{r}{i}^{\ast }=\underset{i}{\mathrm{argmax}}\sum _{i=1}^M\log P_{}(x_i)=\underset{i}{\mathrm{argmax}}\sum _{i=1}^M\log \sum _{k=1}^K{P}_{}(x_i,c_k)\end{array}$$
There is no way to directly calculate θ \theta from the above formula.$\theta$ 's. Therefore, some special skills are needed. We first scale this equation as follows:
$$\begin{array}{rl}\sum _{i=1}^M\log \sum _{k=1}^K{P}_{}(x_i,c_k)=& \sum _{i=1}^M\log \sum _{k=1}^{K}Q(c_k)\frac{P_{}(x_i,c_k)}{Q\left(c_k\right)}\\ \ge & \sum _{i=1}^M\sum _{k=1}^{K}Q\left(c_k\right)\log \frac{P_{}(x_i,c_k)}{Q\left(c_k\right)}\end{array}$$
The above equation (3) introduces an unknown new distribution, $Q\left(c\right)$ , Jensen's inequality is used in equation (4):
$$\begin{array}{r}\log \sum _j{q}_j{x}_j\ge \sum _j{q}_j\log x_j\end{array}$$
Specifically, since the logarithmic function is a concave function, there is:
$$\begin{array}{r}f(\mathbb{E}(x))\ge E\left(f\right(x)),\; iff\left(x\right)is\; a\; concave\; function\; (if\; it\; is\; a\; convex\; function,\; then:\le ）\end{array}$$
At this time, if we want to satisfy the equality sign of Jensen's inequality, we have:
$$\begin{array}{r}\frac{P_{}(x_i,c_k)}{Q(c_k)}=t,tis\end{array}$$
Since $Q\left(c\right)$ is a distribution, so:
$$\begin{array}{r}\sum _{k=1}^{K}Q\left(c_k\right)=1\end{array}$$
Additionally, we have the following equation:
$$\begin{array}{r}Q(c_k)=\frac{P_{}(x_i,c_k)}{{\sum }_{k=1}^K{P}_{}(x_i,c_k)}=\frac{P_{}(x_i,c_k)}{P_{}\left(x_i\right)}=P_{}(c_k|x_i)\end{array}$$
Therefore, if $Q\left(c_k\right)=P_{}(c_k|x_i)$ , then equation (4) is a lower bound of our log-likelihood containing hidden data. If we can maximize this lower bound, that means we can maximize our log-likelihood. That is, we need to maximize the following formula:
$$\begin{array}{r}\underset{i}{\mathrm{argmax}}\sum _{i=1}^M\sum _{k=1}^{K}Q(c_k)\log \frac{P_{}(x_i,c_k)}{Q\left(c_k\right)}\end{array}$$
Remove the constant part in the above formula, because in the above formula $0\le Q\left(c_k\right)\le 1$ , then the lower bound of the log likelihood we need to maximize is,
$$\begin{array}{r}\underset{i}{\mathrm{argmax}}\sum _{i=1}^M\sum _{k=1}^{K}Q(c_k)\log P_{}(x_i,c_k)\end{array}$$
The above formula is the M step of our EM algorithm. What about the E step? Note that Q ( ck ) Q(c_k) in the above formula$Q\left(c_k\right)$ is a distribution, so${\sum }_{k=1}^{K}Q\left(c_k\right)\log P_{}(x_i,c_k)$ can be understood as$\log P_{}(x_i,c_k)$ based on conditional probability distribution$Q(c_k)$ an expectation.

So far, we understand the specific mathematical meaning of the E step and M step in the EM algorithm.

3. EM algorithm process

• enter
  • Observed data: X = { x 1 , x 2 , x 3 , . . . x M }X={ x1​,x2​,x3​,... xM​}
  • Joint distribution: $P_{}(x,c)$
  • conditional distribution
  • Maximum number of iterations: $S$
• Main process
  • 1) Randomly initialize model parameters θ \thetaInitial value of$\theta$$i^0$
  • 2) EM algorithm iteration (V times)
    • E:Specify the equation for the equation:
      $$\begin{array}{r}Q(c_k)=P_{i^s}(c_k|x_i)\\ L(\theta ,i^s)=\sum _{i=1}^M\sum _{k=1}^{K}Q\left(c_k\right)\log P_{}(x_i,c_k)\end{array}$$
    • M:Constant $L(\theta ,i^s)$, get$i^{s+1}$ :
      $$\begin{array}{r}{i}^{s+1}=\underset{i}{\mathrm{argmax}}L(\theta ,i^s)\end{array}$$
    • result $i^{s+1}$ has converged, and the algorithm ends. Otherwise continue iteration.
• output
  • Model parameters $i$

4. Two questions

4.1 How does the EM algorithm ensure convergence?

To prove that the EM algorithm converges, we need to prove that the value of our log-likelihood function has been increasing during the iteration process. That is:
$$\begin{array}{r}\log P_{i^{s+1}}(x_i)\ge \log P_{i^s}(x_i)\end{array}$$
For example,
$$\begin{array}{r}L(\theta ,i^s)=\sum _{i=1}^M\sum _{k=1}^K{P}_{i^s}(c_k|x_i\left)\log P_{}\right(x_i,c_k)\dots \dots Note:Q(c_k)=P_{i^s}(c_k|x_i)\end{array}$$
For example:
$$\begin{array}{r}H(i,i^s)=\sum _{i=1}^M\sum _{k=1}^K{P}_{i^s}(c_k|x_i)\log P_{}(c_k|x_i)\end{array}$$
Determine the equation:
$$\begin{array}{rl}L(\theta ,i^s)-H(i,i^s)=& \sum _{i=1}^M\sum _{k=1}^K{P}_{i^s}(c_k|x_i)\log \frac{P_{}(x_i,c_k)}{P_{}(c_k|x_i)}\\ =& \sum _{i=1}^M\sum _{k=1}^K{P}_{i^s}(c_k|x_i\left)\log P_{}\right(x_i)\dots \dots P_{i^s}(c_k|x_i)is\; known,\; and\; the\; sum\; is1\\ =& \sum _{i=1}^M\log P_{}\left(x_i\right)\end{array}$$
In the final formula of the above formula, $\theta$ takes$i^s$ 和 $i^{s+1}$ ,theoretically:
$$\begin{array}{rl}& \sum _{i=1}^M\log P_{i^{s+1}}(x_i)-\sum _{i=1}^M\log P_{i^s}\left(x_i\right)\\ =& \left[L\right(i^{s+1},i^s)-L(i^s,i^s)]-\left[H\right(i^{s+1},i^s)-H(i^s,i^s)]\end{array}$$
To prove the convergence of the EM algorithm, we only need to prove that the right side of the above equation is non-negative. Since $i^{s+1oneL}$ ( θ , θ s ) L(\theta, \theta^s)$L(\theta ,i^s)$Specific, slope:
$$\begin{array}{r}L(i^{s+1},i^s)-L(i^s,i^s)\ge 0\end{array}$$
Let us give the equation:
$$\begin{array}{rl}H(i^{s+1},i^s)-H(i^s,i^s)=& \sum _{i=1}^M\sum _{k=1}^K{P}_{i^s}(c_k|x_i)\log \frac{P_{i^{s+1}}(c_k|x_i)}{P_{i^s}(c_k|x_i)}\\ \le & \sum _{i=1}^M\log \sum _{k=1}^K{P}_{i^s}(c_k|x_i)\frac{P_{i^{s+1}}(c_k|x_i)}{P_{i^s}(c_k|x_i)}\\ =& \sum _{i=1}^M\log \sum _{k=1}^K{P}_{i^{s+1}}(c_k|x_i)=0\end{array}$$
Among them, equation (25) uses Jensen's inequality, which is just the opposite of the use in Section 2. Type (26) uses the property that the probability distribution accumulates to 1. At this point, we get:
$$\begin{array}{r}\log P_{i^{s+1}}(x_i)-\log P_{i^s}\left(x_i\right)\ge 0\end{array}$$
This further proves the convergence of the EM algorithm.

4.2 If the EM algorithm converges, is it guaranteed to converge to the global maximum?

It is not difficult to see from the above derivation that the EM algorithm can guarantee convergence to a stable point, but it cannot guarantee convergence to the global maximum point, so it is a local optimal algorithm. Of course, if our optimization goal L ( $L(\theta ,i^s)$is convex, then the EM algorithm can ensure convergence to the global maximum, which is the same as iterative algorithms such as the gradient descent method.

5. Summary

If we think about the EM algorithm from the perspective of algorithmic thinking, we can find:

• What is known in the algorithm is the observation data,
• What is unknown are the underlying data and model parameters,
• In step E, what we do is to fix the values ​​of the model parameters and optimize the distribution of the hidden data.
• In step M, what we do is to fix the implicit data distribution and optimize the values ​​of model parameters.

Comparing other machine learning algorithms, in fact, many algorithms have similar ideas. For example, the SMO algorithm (Support Vector Machine Principle (4) SMO Algorithm Principle) and the coordinate axis descent method (Lasso Regression Algorithm: Summary of Coordinate Axis Descent Method and Minimum Angle Regression Method) all use similar ideas to solve problems.

reference

【1】http://www.cnblogs.com/pinard/p/6912636.html