[Introduction to Machine Learning] Cross-entropy loss function and MLE criterion

0 summary

The conclusion is released first for easy viewing, and the detailed derivation can be read in subsequent chapters.

0.1 MLE and cross entropy

The loss function derived by the MLE criterion is equivalent to the cross-entropy loss function, as follows:
$$J(i\text{i}i)=-{\mathbb{E}}_{x\text{x}x,y\text{y}y\sim P_{data}}[log{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}\theta )]$$
Supervised learning is given input$x\text{x}x$ (sample) and output$y\text{y}y$ (label) to train the model.

• From the perspective of the maximum likelihood estimation criterion , this process can be regarded as the conditional probability $P(Y|X;i\text{i}$A maximum likelihood estimation process of$\theta$$)$
• From the perspective of relative entropy and cross entropy , this process can be seen as adjusting $Q\left(X\right)$ makes it approximate$The\; process\; of\; P\left(X\right)$ , and$H(P,The\; minimum\; value\; of\; Q)$ is$H\left(P\right)$ , then we have the cross entropy$H(P,Q\; (\; X)$ Q(X)obtained by minimizing$Q\left(X\right)$ should be able to approximate$P(X)$。

See Section 4 for details

0.2 Specific form of cross entropy loss function

Among the following symbols: $N$ is the number of samples,$M$ is the number of categories

1. Linear regression problem

$$\begin{array}{rl}J(i\text{i}i)& =-{\mathbb{E}}_{x\text{x}x,y\text{y}y\sim P_{data}}[log{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}i)]\\ & =\frac{1}{N}||y\text{y}y-\hat{y}\text{y^}\hat{y}|{|}_2^2\end{array}$$

2.Logistic regression (two classification problem)

$$J(i\text{i}i)=\frac{1}{N}\sum _{i=1}^N-y_{i}log\widehat{y_i}-(1-y_i)log(1-\widehat{y_i})$$ where:

• $y_i$Is 0 or 1, if the category is A, it is 1, if the category is B, it is 0
• $\widehat{y_i}$is the probability of classifying the sample as A output by the model, then $(1-\widehat{y_i})$ is the probability of being classified as B,$\widehat{y_i}=p_{sigmoid}({i\text{i}i}^T{x}_i\text{xi}{x}_i+b)$

3.Multiple classification problems

$$J(i\text{i}i)=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^M{y}_{i,j}log{\hat{y}}_{i,j}$$in:

• $y_{i,j}$Is 0 or 1, if category $j$ fits sample$i$ takes the value 1, otherwise it takes the value 0
• ${\hat{y}}_{i,j}$For the model output, sample $i$ is classified into category$j$的概率，${\hat{y}}_{i,j}=p_{softmax}([{i\text{i}i}^T{x}_i\text{xi}{x}_i+b\text{b}b{]}_j)$
• $[{i\text{i}i}^T{x}_i\text{xi}{x}_i+b\text{b}b{]}_j$is the jjth output layer$j$ outputs, interpretable as categoriesjjlog probability of$j$

1 Maximum likelihood estimation MLE

1.1 Likelihood function and log-likelihood function

Given a probability distribution $P\left(x\right)$ , assuming that the distribution is governed by a certain set of parameters$i\text{i}\theta$ is determined, then the probability distribution can be written as$P(x;i\text{i}\theta )$form.

• Change the parameter $i\text{i}\theta$ is fixed, and$If\; x$ is regarded as a variable, then$P(x;i\text{i}\theta )$is called the probability distribution, which can be regarded as the parameter$i\text{i}$The specific probability distribution when$\theta \; takes\; a\; certain\; set\; of\; specific\; values.$
• Will $x$ is fixed, and$\theta$ is regarded as a variable, then$L(i\text{i}i)=P(x;i\text{i}\theta )$is called the likelihood function. The likelihood function can be viewed astheWhen$x$$\theta$ under different values$The\; probability\; of\; x$ happening.

For a set of independent and identically distributed data $x\text{x}x=(x_1,x_2,...,x_n{)}^T$ , its joint distribution can usually be written in the form of a continuous product:$$L\left(i\text{i}i\right)=P(x\text{x}x;i\text{i}i)=\prod _{i=1}^{n}P(x_i;i\text{i}The\; more\; commonly\; used\; form\; of\; \theta )$$is to take the logarithm of the likelihood function, and the continuous multiplication can be converted into a summation symbol. This function is called the log-likelihood function: LL ( θ ) =$$LL(i\text{i}i)=logP(x\text{x}x;i\text{i}i)=\sum _{i=1}^{n}logP(x_i;i\text{i}\theta )$$Since the logarithmic function is a monotonically increasing function, taking the logarithm will not affect the position of the extreme value.

1.2 Maximum Likelihood Estimation

As mentioned above "the likelihood function can be viewed as theWhen$x$$\theta$ under different values$The\; probability\; of\; x$ happening. "Then the idea of ​​the maximum likelihood estimation criterion is:
I want to estimate$\theta$ , since we already have such a set of observation data$x\text{x}x=(x_1,x_2,...,x_n{)}^T$ , then maximize the probability of its occurrence, and then take the corresponding$\theta$ is good as an estimate.
Expressed as a mathematical formula:$${\widehat{i\text{i}i}}_{ML}=\underset{i}{\mathrm{argmax}}L(i\text{i}i)=\underset{i}{\mathrm{argmax}}\sum _{i=1}^{n}logP(x_i;i\text{i}\theta )$$
After averaging the above formula, it can be equivalently written as:
$${\widehat{i\text{i}i}}_{ML}=\underset{i}{\mathrm{argmin}}[-\mathbb{E}[logP(x_i;i\text{i}\theta )\left]\right]$$
In the process of parameter estimation using the maximum likelihood criterion, the likelihood function is used to estimate the parameter$i\text{i}\theta$ is differentiated and then the maximum value is obtained. The solution process will not be described here.
As can be seen from this overview of the solution process, using the data$x\text{x}x$ solves for$i\text{i}The\; premise\; of\; \theta$ is the function$P(x;i\text{i}\theta )$is correct, otherwise the function cannot correctly reflect$x\text{x}x$ and$i\text{i}The\; real\; relationship\; of\; \theta$ , then for$i\text{i}$It is impossible to estimate$\theta \; .$This property is described in "Deep Learning" as follows:

• True distribution $p_{data}$Must be in the model family $p_{model}(\cdot ;i\text{i}\theta )$. Otherwise, there is no estimate to restore$p_{data}$。
• True distribution $p_{data}$Must correspond to exactly one $i\text{i}\theta$ value. Otherwise, the maximum likelihood estimate recovers the true distribution$p_{data}$Finally, it is impossible to decide which $i\text{i}i$ .

In layman's terms, the shape of the function must be chosen correctly, and which parameters should be chosen as well.

2 Relative entropy and cross entropy

2.1 Relative entropy

Relative entropy, also known as Kullback-Leibler divergence or information divergence, is an asymmetric measure of the difference between two probability distributions. In information theory, relative entropy is equivalent to the difference in information entropy of two probability distributions. ——Baidu Encyclopedia "Relative Entropy"

Definition of relative entropy: $$\begin{array}{rl}D(P||Q)& ={\mathbb{E}}_{x\sim P(X)}[log\frac{P(x)}{Q(x)}]\\ & =\sum _{x\in X}P(x)log\frac{P\left(x\right)}{Q\left(x\right)}\\ & =\sum _{x\in X}P(x)logP(x)-\sum _{x\in X}P(x)logQ(x)\end{array}$$where ${\mathbb{E}}_{x\sim P(X)}$means that it obeys $P\left(X\right)$ distribution of$To\; calculate\; the\; mathematical\; expectation\; of\; x$ , let$$\begin{gathered} H(P)=-\sum _{x\in X}P(x)logP(x) \\ H(P,Q)=-\sum _{x\in X}P\left(x\right)logQ\left(x\right) \end{gathered}$$ relative entropy can be written in the following form:$$D(P||Q)=H(P,Q)-H\left(P\right)$$
When the logarithm is based on base 2, the formulas and symbols have the following physical meanings:

• $H\left(P\right)$ is subject toP (X) P(X)SourceXX$of\; P$$($$X$$)\; distribution$The information$entropy$ ofAverage number of bits required for$X\; encoding$
• $H(P,Q)$ is calledcross entropy, which means that the pair obeysP ( X ) P(X)SourceXX$of\; P$$($$X$$)\; distribution$$X$ , according to the distribution$Q(\_\_$ _$Q\left(X\right)$ means obeyingP (X) P(X)XX$of\; P$$($$X$$)$ distribution$X$ degree of difficulty
• $D(P||Q)$ is the relative entropy, expressed as$Q\left(X\right)$ comes to sourceXXThe average number of extra bits required for$X\; encoding$

2.2 Cross entropy

The definition of cross entropy is derived from relative entropy above:
$$\begin{array}{rl}H(P,Q)& =-\sum _{x\in X}P(x)logQ(x)\\ & =-\mathbb{E}[logQ(x)]\end{array}$$Obviously, when the distribution $Q\left(X\right)$ approachesP (X) P(X)When$P$$($$X$$)$$H(P,Q)$ tends to$H\left(P\right)$ , relative entropy$D(P||Q)$ tends to 0, which means using$Q\left(X\right)$ coding also does not have many redundant bits.

It can be proved that $D(P||Q)\ge 0$ , that is,$minH(P,Q)=H(P)$。

In machine learning, our model training is actually a process of parameter estimation. Our process of adjusting the parameters of the model is to adjust the model $Q\left(X\right)$ to approximate the real data$The\; optimization\; process\; of\; P\left(X\right)$ .

3 Give an example of a classifier

The cross-entropy loss function is often used in classification problems. The following is an example of classification.

• There are N input samples $x_i\text{xi}{x}_i,i=1,2,...,N$ (for example: N pictures)
• There are M categories $y_j,j=1,2,...,M$ (for example: cats, dogs and other M animals)
• $P_{data}(y_j|x_i\text{xi}{x}_i)$ is the distribution to be learned, and one-hot encoding is generally used in multi-classification (for example:$P_{data}(cat|catphoto)\_=1,P_{data}(photosofdogsandcats)=0$
• $P_{model}(y_j|x_i\text{xi}{x}_i;i\text{i}\theta )$P data ( yj ∣ xi ) P_{data}(y_j|\pmb{x_i})through learning$P_{data}(y_j|x_i\text{xi}{x}_i)$ )
  According to the above description, item$The\; cross\; entropy\; of\; i$ samples can be expressed as:$$H_i(P_{data},P_{model})=-\sum _{j=1}^M{P}_{data}(y_j|x_i\text{xi}{x}_i)log{P}_{model}(y_j|x_i\text{xi}{x}_i;i\text{i}\theta )$$so that we can derive the cross-entropy loss function of this sample:
  $$\begin{array}{rl}{J}_i(i\text{i}i)& =H_i(P_{data},P_{model})\\ & =-\sum _{j=1}^M{P}_{data}(y_j|x_i\text{xi}{x}_i)log{P}_{model}(y_j|x_i\text{xi}{x}_i;i\text{i}i\end{array}$$
  In fact, when using one-hot encoding, only one item in the summation sign is non-zero at a time. For example: Suppose $P_{data}=(cat,dog,mouse){}^T$ , then given a photo of a dog, the probability distribution of three animals: cat, dog, and mouse is:
  $$P_{data}(y=cats,dogs,mice|\_x_i\text{xi}{x}_i=dogphotos)\_=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]$$The hypothesis of the inference result of the model is:
  $$P_{model}(y=cats,dogs,mice|\_x_i\text{xi}{x}_i=dogphotos;\_i\text{i}i)=\left[\begin{array}{c}0.2\\ 0.6\\ 0.2\end{array}\right]$$Then the actual result of the cross entropy loss function of this sample is:
  $$\begin{array}{rl}{J}_i(i\text{i}i)& =-\sum _{y_j=cat,dog,mouse}{P}_{data}(y_j|x_i\text{xi}{x}_i=dogphoto)logP\_{}_{model}(y_j|x_i\text{xi}{x}_i=dogphotos;\_i\text{i}i)\\ & =-P_{data}(y=dog|\_x_i\text{xi}{x}_i=dogphoto)logP\_{}_{model}(y=dog|\_x_i\text{xi}{x}_i=dogphotos;\_i\text{i}i)\\ & =-1\cdot log0.6\end{array}$$

---

From this , the calculation formula of the cross-entropy loss function in multi-classification problems can be derived $$J_i(i\text{i}i)=-\sum _{j=1}^M{y}_{j,i}log{P}_{j,i}$$

• $y_{j,i}$Is 0 or 1, if category $j$ fits sample$i$ takes the value 1, otherwise it takes the value 0
• $P_{j,i}$For the model output, sample $i$ is classified into category$The\; probability\; of\; j$ can be obtained using the softmax unit

When the classification problem is a binary classification , it can be simplified to: $$J_i(i\text{i}i)=-y_{i}log{P}_i-(1-y_i)log(1-P_i)$$

• $y_i$Is 0 or 1, if the category is A, it is 1, if the category is B, it is 0
• $P_i$is the probability of classifying the sample as A output by the model, then $(1-P_i)$ is the probability of being classified as B, which can be obtained by using the sigmoid unit

4 Cross-entropy loss function and MLE criterion

When we specify the output of a model $P_{model}(y\text{y}y|x\text{x}When\_$_
$$J(i\text{i}i)=-\mathbb{E}[log{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}\theta )]$$The mathematical expectation (mean) is for the training data distribution$P_{data}$of samples and labels, so it can be written as $$J(i\text{i}i)=-{\mathbb{E}}_{x\text{x}x,y\text{y}y\sim P_{data}}[log{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}\theta )]$$
Supervised learning is given input$x\text{x}x$ (sample) and output$y\text{y}y$ (label) to train the model.

• From the perspective of the maximum likelihood estimation criterion , this process can be regarded as the conditional probability $P(Y|X;i\text{i}$A maximum likelihood estimation process of$\theta$$)$
• From the perspective of relative entropy and cross entropy , this process can be seen as adjusting $Q\left(X\right)$ makes it approximate$The\; process\; of\; P\left(X\right)$ , and$H(P,The\; minimum\; value\; of\; Q)$ is$H\left(P\right)$ , then we have the cross entropy$H(P,Q\; (\; X)$ Q(X)obtained by minimizing$Q\left(X\right)$ should be able to approximate$P(X)$。

Different models will have different $P_{model}(y\text{y}y|x\text{x}x)$, therefore different specific loss function shapes will be derived.

4.1 Linear regression

Given feature $h\text{h}h$ , linear unit output$\hat{y}\text{y^}\hat{y}={i\text{i}i}^{T}h\text{h}h+b\text{b}b$ , the model probability is$P_{model}(y\text{y}y|x\text{x}x;i\text{i}i)=\mathcal{N}(y\text{y}y;\hat{y}\text{y^}\hat{y},I\text{I}I)$，即有
$$\begin{array}{rl}{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}i)& =\frac{1}{{\sqrt{2p.m}}^N}{e}^{-\frac{1}{2}(y\text{y}y-\hat{y}\text{y^}\hat{y}{)}^T(y\text{y}y-\hat{y}\text{y^}\hat{y})}\\ & =\frac{1}{{\sqrt{2p.m}}^N}{e}^{-\frac{1}{2}{\sum }_{i=1}^N(y_i-{\hat{y}}_i{)}^2}\end{array}$$
其者对数似然第二的：
$$\begin{array}{rl}-ln{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}i)& =\frac{N}{2}ln2\pi +\frac{1}{2}\sum _{i=1}^N(y_i-{\hat{y}}_i{)}^2\\ & =\frac{N}{2}ln2\pi +\frac{1}{2}||y\text{y}y-\hat{y}\text{y^}\hat{y}|{|}_2^2\end{array}$$
The cost function can be derived (ignoring the constant terms and coefficients $\frac{1}{2}$）：
$$\begin{array}{rl}J(i\text{i}i)& =-{\mathbb{E}}_{x\text{x}x,y\text{y}y\sim P_{data}}[log{P}_{model}(y\text{y}y|x\text{x}x;i\text{i}i)]\\ & =\frac{1}{N}||y\text{y}y-\hat{y}\text{y^}\hat{y}|{|}_2^2\end{array}$$
Obviously, the cost function is MSE, so for linear regression problems, the specific cost function derived from the maximum likelihood estimation criterion and the cross-entropy loss function is MSE.

4.2 Logistic regression (two classification problem)

The sigmoid function can compress the output data to the (0,1) interval, which can be interpreted as probability. The function form is as follows $$\sigma \left(z\right)=\frac{1}{1+e^{-z}}$$The third example is as follows
$$\begin{array}{rl}J(i\text{i}i)& =\frac{1}{N}\sum _{i=1}^N{J}_i(i\text{i}i)\\ & =\frac{1}{N}\sum _{i=1}^N-y_{i}log{P}_{model}(1|x_i\text{xi}{x}_i;i\text{i}i)-(1-y_i)log{P}_{model}(0|x_i\text{xi}{x}_i;i\text{i}i)\\ & =\frac{1}{N}\sum _{i=1}^N-y_{i}log\widehat{y_i}-(1-y_i)log(1-\widehat{y_i})\end{array}$$in:

• $y_i$Is 0 or 1, if the category is A, it is 1, if the category is B, it is 0
• $\widehat{y_i}$is the probability of classifying the sample as A output by the model, then $(1-\widehat{y_i})$ is the probability of being classified as B,$\widehat{y_i}=s({i\text{i}i}^T{x}_i\text{xi}{x}_i+b)$

4.3 Multi-classification problem

The softmax function is a generalization of the sigmoid function. The function form is as follows $$s(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^M{e}^{z_j}}$$From the example in Section 3, we can see that the cost function is
$$\begin{array}{rl}J(i\text{i}i)& =\frac{1}{N}\sum _{i=1}^N{J}_i(i\text{i}i)\\ & =-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^M{y}_{i,j}log{P}_{model}(j|x_i\text{xi}{x}_i;i\text{i}i)\\ & =-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^M{y}_{i,j}log{\hat{y}}_{i,j}\end{array}$$in:

• $y_{i,j}$Is 0 or 1, if category $j$ fits sample$i$ takes the value 1, otherwise it takes the value 0
• ${\hat{y}}_{i,j}$For the model output, sample $i$ is classified into category$j$的概率，${\hat{y}}_{i,j}=s\left(\right[{i\text{i}i}^T{x}_i\text{xi}{x}_i+b\text{b}b{]}_j)$
• $[{i\text{i}i}^T{x}_i\text{xi}{x}_i+b\text{b}b{]}_j$is the jjth output layer$j$ outputs, interpretable as categoriesjjlog probability of$j$