Machine Learning and High-Dimensional Information Retrieval - Note 2 - Statistical Decision Making and Machine Learning

2. Statistical decision-making and machine learning

The basic problem is that for a random variable $X\in {\mathbb{R}}^{}$For some observations in${}^p$ , we want to obtain the random variable YYThe "most likely" value of$Y.$For simplicity, we assume$Y$ inR \mathbb{R}Implementation in$R.$We further assume that the resulting joint probability density$p_{X,Y}(x,y)$ has been given.

Example 2.1
$X$ is a (vector) image of a person. In Figure 2.1 below, this observation represents an image containing only one pixel, which has varying degrees of gray.

$$Y=\{\begin{array}{ll}1:& X\text{ is picture of person }\mathrm{A}\\ 0:& X\text{ is not }\end{array}$$

Its purpose is to predict given $X$的$Y.$ _ If there is a person calledA \mathrm{A}Person$A\; ,\; he\; mainly\; wears\; dark\; clothes$

Figure 2.1: Two discrete random variables $X$和YYJoint probability distribution of$Y$

It is reasonable to believe that the respective probabilities of events are expressed as: $p_{42}>p_{32}>p_{22}>p_{12}$ 和$p_{11}>p_{21}>p_{31}>p_{41}$.

To determine what "most likely" means, we consider the squared loss function ${}^1$

${}^{1Note}$ that other loss functions are possible and do make sense, the squared distance was chosen simply for convenience reasons.

$$L(Y,f(X))=(Y-f(X){)}^2.\text{\hspace{1em}(2.1)}$$

We want to choose $f$ , making the expected value of the loss function, the so-called expected prediction error
$$EPE\left(f\right)=\mathbb{E}[L(Y,f\left(X\right)\left)\right]$$
is the smallest. A simple calculation using the tools described in subsection 1.2 gives

$$\begin{array}{rl}EPE\left(f\right)& =\mathbb{E}[L(Y,f(X))]=\mathbb{E}\left[(Y-f(X){)}^2\right]\\ & ={\int }_{{\mathbb{R}}^p\times \mathbb{R}}(y-f(x){)}^2{p}_{X,Y}(x,y)\mathrm{d}x\mathrm{d}y\\ & ={\int }_{R^p}\underset{=:{\mathbb{E}}_{Y\mid X=x}\left[(Y-f(X){)}^2\right]}{\underbrace{{\int }_{\mathbb{R}}(y-f(x){)}^2{p}_{Y\mid X=x}(y)d}}{p}_X(x)\mathrm{d}x\\ & ={\mathbb{E}}_X{E}_{Y|X=x}\left[(Y-f(X){)}^2\right]\end{array}$$
因此$f(X)=\underset{c}{\mathrm{argmin}}{\mathbb{E}}_{Y\mid X=x}\left[(Y-c{)}^2\right]$. SinceE [ ⋅ ] \mathbb{E}[\cdot]linearity of$E$$[$$\cdot$$]\; ,\; which\; can\; be\; simplified\; to$

$$f(X)=\underset{c}{\mathrm{argmin}}\left({\mathbb{E}}_{Y|X=x}\left[Y^2\right]-2c{\mathbb{E}}_{Y|X=x}[Y]+c^2\right).\text{\hspace{1em}(2.2)}$$

The quadratic term is easily minimized, and the optimal value is
$$f(X)={\mathbb{E}}_{Y\mid X=x}[Y].\text{\hspace{1em}(2.3)}$$

Theorem 2.2

At given $In\; the\; case\; of\; X$ , if the best prediction is measured as the square of the loss, forYYThe best predictor of$Y\; is\; the\; conditional\; mean.$

As mentioned above, the conditional mean as the best predictor relies on the fact that we use the squared loss as a quality measure. We can show that if we take the absolute value $|Y-f(\_\_\_$ _${\ell }_1$Loss, the best prediction is the conditional median. While a rigorous proof of this statement is beyond the scope of this lecture note, we can easily see that the median is empirically ${\ell }_1$The optimal value that minimizes the loss

$$\underset{c}{\mathrm{argmin}}\frac{1}{n}\sum _i\left|y_i-c\right|.\text{\hspace{1em}(2.4 )}$$

We only need the result of a non-smooth convex optimization, that is, for a convex function with subgradient, if and only if $c^{}$c ∗ c^{*}can only be achieved when the sub-gradient at${}^{\ast \; contains\; 0}$$c^{The\; minimum\; value\; of\; \ast }$ . In our case, the experience${\ell }_1$loss relative to ccThe sub-gradient of$c$$\frac{1}{n}{\sum }_i\mathrm{sign}\left(y_i-c\right)$ for$c\ne y_i$ 与$\left\{\frac{1}{n}{\sum }_{i\ne j}\mathrm{sign}\left(y_i-c\right)+t\mid -1\le t\le 1\right\}$ for$c=y_j$. Therefore, we see that only if it is less than $c$的$y_i$The number of yi y_{i} greater than c$y_i$When the quantities of are the same, ccOnly the sub-gradient at$c\; contains\; zero.$in odd number$In\; the\; case\; of\; n$ ,$c$ must be the same as$y_i$( $(n+The\; maximum\; value\; of\; 1)/2$ coincides with the maximum value. This is exactly the definition of median.

2.1 General settings for supervised decision-making and generalization error

Let us resume the above chapters from a higher level perspective. We introduce a loss function based on training samples $\left(x_i,y_i\right),i=1,\dots ,N$ , can be obtained from a given function classF \mathcal{F}Learning the best prediction functionff$in\; F$$f$ . Such methods are called supervised learning methods because they require a training set with each sample$x_i$There is a corresponding $y_i$. A general problem statement is:

让$(X,Y)\in {\mathbb{R}}^{}$Random variables in${}^p$$F$ comes from${\mathbb{R}}^p\Rightarrow$A class of functions in$R.$These functions can be parameterized with limited parameters, such as$Th\in R^{M.}$ _ Furthermore, let$L:R\times R\to R_0^{+}$is a loss function used to measure $Y$ andf ( X ) f(X)Deviation of$f$$($$X$$)\; .$The purpose of supervised prediction methods is to predict when$\hat{f}\in$Find the minimum expected prediction errorE [ L ( Y , f ( X ) ] \mathbb{E}[L(Y, f(X)]$in\; F$$\mathbb{E}[L(Y,f(X)]$。

In fact, in order to build a supervised machine learning method, we need to figure out three questions.

• We have to specify the loss function $L$。

• We must specify the function class $\mathcal{F}$。

• Since in practice we don't know $(X,Y)$ , we need training samples$\left(x_i,y_i\right),i=1,\dots ,N$ to approximate the expected prediction error.

So we get an optimization problem

$$\hat{f}=\underset{f\in \mathcal{F}}{\mathrm{argmin}}\frac{1}{N}\sum _{i=1}^{N}L\left(y_i,f\left(x_i\right)\right)\text{\hspace{1em}(2.5)}$$

These optimization problems are more or less difficult to solve, but in principle they can be fed into the optimization toolbox in the form described above.

Example: Linear Regression
For linear regression, we choose $L$ is the quadratic loss, that is,$L(Y,f(X))=(Y-f(X){)}^2$ , we choose$F$ is an affine function

$$f(x)=i_0+\sum _{k=1}^p{i}_k{x}^{(k)},\text{\hspace{1em}(2.6)}$$

where $x^{\left(k\right)}$ is the vector$entry\; for\; x$ . Considering$N$ training samples, then the optimization problem is

$$\widehat{Th}=\underset{i_0,...,i_p}{\mathrm{argmin}}\frac{1}{N}\sum _{i=1}^N{\left(y_i-i_0-\sum _{k=1}^p{i}_k{x}_i^{(k)}\right)}^2.\text{\hspace{1em}(2.7)}$$

Definition :
For a given function $f$ , the difference between experienced loss and expected loss

$$G_N(f)=\mathbb{E}\left[L(Y,f(X)\right]-\frac{1}{N}\sum _{i=1}^{N}L\left(y_i,f\left(x_i\right)\right).\text{\hspace{1em}(2.8)}$$

called $The\; generalization\; error\; of\; f$ .

According to the law of large numbers, we know that $G_N$in $N$ approaches 0 as it approaches infinity. However, many times the number of training samples is limited. In addition, the speed of convergence is highly dependent on$(X,$The (unknown) probability distribution of$Y$$)\; .$Therefore, one goal of machine learning is to constrain the generalization error with high probability.

In practice we use cross validation (cross entropy) to check $How\; well\; f$ matches the data.

2.2 k nearest neighbors

In practice, the problem we face is that we do not know $X$和$joint\; distribution\; of\; Y$ , so we must estimate the conditional mean in equation (2.3). Typically this will be achieved by given a specific$X$的YYto complete the relative frequency of$Y.$However, in the continuous case, problems arise, even in many observations$\left(x_i,y_i\right)$ , there may not be$x_i=x$ . This problem can be expanded byxxarea around$x$$x_i$Close to expectations $to\; solve\; for\; the\; observation\; of\; x$ . This leads to$The\; concept\; of\; k$ nearest neighbors.

Figure 2.2: Given the events $(1,1),(2,5),(3,3),(4,1),(5,4)$ Binary distribution$(X,Y)$。虽然${\mathbb{E}}_{Y\mid X=4}[Y]=5$ , averaging over 3 nearest neighbors provides${\hat{f}}_{k=3}（4）=（2+3+5）/3=10/3$。

$The\; k$ nearest neighbor method estimates$f$：

$${\hat{f}}_k(x)=\text{ average }\left(y_i\mid x_i\in N_k(x)\right)\text{, }\text{\hspace{1em}(2.9)}$$

Among them, $N_k\left(x\right)$ meansxxkkof$x$$The\; set\; of\; k$ nearest neighbors. There are two approximations here.

1. Expectations are approximated by averaging.

2. Conditions at a point are approximated by conditions in some region (around the point).

If $N$ is the number of observations, so pay attention to the following points. If$N$ increases,$x_i$will be close to $x$ . Furthermore, if$As\; k$ increases, the average will tend to the expected value. More precisely, under mild conditions of joint probability measurement, we have

$$\underset{N,k\to \infty ,\frac{k}{N}\to 0}{\lim }\hat{f}(x)={\mathbb{E}}_{Y\mid X=x}[Y].\text{\hspace{1em}(2.10)}$$

However, the sample size $N$ is usually limited, so the sample size is not sufficient to satisfy the condition.

There are two ways to overcome this problem.

• to $f$ imposes model assumptions, for example,$f(x)\approx x^T\beta$gets linear regression.

• By "lowering" XXDimensions of$X.$This leads to the motivation of dimensionality reduction.

找到$f:\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right]\mapsto \left[\begin{array}{c}{s}_1\\ ⋮\\ s_k\end{array}\right]$, make $f$ maintains the "intrinsic" distance of the observation.

2.3 Curse of Dimension

让$X\in {\mathbb{R}}^p$ is a random variable. There are a few things to observe.

First of all, XXThe absolute$noise$$p$ ) increases because noise accumulates in all dimensions.

Second, estimate XXThe number of observations required for the densityfunction$of$increases sharply with the increase of$p\; .$The density function is usually estimated with the help of the relative frequency of an occurrence. As an example, assume that a NNin a one-dimensional space$N$ observations need to estimate a certain density function to a predetermined accuracy. If the observation space increases to 2 dimensions, the number of observations must increase to$N^{A\; quantity\; of\; 2}$ is required to maintain the same accuracy. For a 3-dimensional space, approximately$N^3$ observations, and so on. This means that, in order for the density function to be estimated with the same accuracy, the number of measurements required grows exponentially with the dimensions of the measurement space.

One cause of the curse of dimensionality is the empty space phenomenon (Scott), which states that high-dimensional spaces are inherently sparse. Given that the data points are evenly distributed in a 10-dimensional unit sphere, the probability that a point is more than 0.9 from the center is less than 35%. Therefore, the tail in a high-dimensional distribution is much more important than the tail in a one-dimensional distribution. Given a high-dimensional multivariate random variable, having one component in the tail of its distribution is sufficient for the entire sample to lie in the tail of the common density function. Figure 2.3 illustrates the one-dimensional case.

Figure 2.3: For a one-dimensional random variable, most observations are around the mean. This is not the case in higher dimensions.