Machine Learning and High-Dimensional Information Retrieval - Note 8 - Support Vector Machines

Note 8 Support Vector Machines

The idea behind support vector machines is very simple. In the simplest case, we assume that the samples of the two classes can be linearly separated, that is, we assume that there is a two-dimensional affine hyperplane that can separate the two classes. SVM is a supervised learning algorithm that can find the "best" separating hyperplane given training samples from two classes. Once you find it, it's easy to classify new data points. Depending on which side of the hyperplane the incoming new data point lies, it is assigned to the corresponding category.

8.1 Geometric basis of SVM

For some $\mathbf{w}\in {\mathbb{R}}^p$ and some nonnegative$b\ge 0$，${\mathbb{R}}^{}$The affine hyperplane in${}^{p\; is\; defined\; as\; the\; set}$

$${\mathcal{H}}_{\mathbf{w},b}:=\left\{\mathbf{x}\in R^p\mid {\mathbf{w}}^{\top }\mathbf{x}-b=0\right\}.\text{\hspace{1em}( 8.1 )}$$
Vector$$\text{\hspace{1em}w mathbf}$$$$\text{\hspace{1em}{}$$$w$ is the normal line, or perpendicular to$H_{w,b}$, because as long as we have an arbitrary line segment x 2 − x 1 \mathrm{x}_{2}-\mathrm{x}_{1} on the hyperplane${\mathrm{x}}_2-x_1$，在$x_1,x_2\in H_{w,b}$,but

$${\mathbf{w}}^{\top }\left({\mathbf{x}}_2-{\mathbf{x}}_1\right)=0.\text{\hspace{1em}(8.2)}$$

From this, it is easy to derive two affine hyperplanes ${\mathcal{H}}_{\mathbf{w},b},H_{\tilde{w},\tilde{b}}$is parallel if and only if $w$是$\stackrel{Multiples\; of\; ~}{w}$ .

${}^1$ This stems from the so-called hyperplane separation theorem, which mainly states that two disjoint convex sets can be separated by a hyperplane. This proof can be traced back to the famous Hermann Minkowski (1864-1909).

Any hyperplane will ${\mathbb{R}}^p$ is completely separated in two half spaces${}^1$ . From a certain point$x$ to${\mathcal{H}}_{\mathbf{w},b}$The signed distance of is defined as

$$d\left(\mathbf{x},H_{w,b}\right)=\frac{{\mathbf{w}}^{\top }\mathbf{x}-b}{\Vert \mathbf{w}\Vert }.\text{\hspace{1em}(8.3)}$$

The reason for this definition is because of the following Lemma.

Lemma $8.1$ Signed distance to affine hyperplanes (Signed distance to affine hyperplanes$)$

from $x$ to$H_{w,b}$The Euclidean distance is given by $|\delta \left(\mathbf{x},{\mathcal{H}}_{\mathbf{w},b}\right)|Decision$ .

Proof.
We use some geometric intuition to prove. Suppose from $Starting\; at\; x$ , we want to follow$r$ direction moves toward the hyperplane. To find the shortest distance we have to solve

$$\min \Vert \mathbf{r}\Vert \text{ s.t. }\mathbf{x}+\mathbf{r}\in H_{w,b}\text{\hspace{1em}(8.4)}$$

The constraints here are equivalent to $(\mathbf{x}+\mathbf{r}{)}^{\top }\mathbf{w}=b$ , or

$${\mathbf{r}}^{\top }\mathbf{w}=b-{\mathbf{x}}^{\top }\mathbf{w}.\text{\hspace{1em}(8.5)}$$

It can be seen from this equation that solving (8.4) for $r$ must be$Multiples\; of\; w$ . We prove this below. Assume$r$ is not$Multiples\; of\; w$ . Then for${\mathbf{w}}_i^{\perp }$Orthogonal to $w$ , we can always write it as$r=\lambda \mathbf{w}+{\sum }_i{m}_i{w}_i^{\perp }$Independently, ${\left(\lambda \mathbf{w}+{\sum }_i{m}_i{\mathbf{w}}_i^{\perp }\right)}^{\top }w=\lambda {\mathbf{w}}^{\top }w$holds, and according to the triangle inequality,$\Vert \mathbf{r}\Vert \ge \Vert \lambda \mathbf{w}\Vert$。

This observation gives rise to the minimization problem,

$$\underset{l}{\min }\Vert \lambda \mathbf{w}\Vert \text{ s.t. }\lambda \Vert \mathbf{w}{\Vert }^2=b-{\mathbf{x}}^{\top }\mathbf{w}\text{\hspace{1em}(8.6)}$$

Its solution is the absolute value of the signed distance

$$\frac{\left|{\mathbf{w}}^{\top }\mathbf{x}-b\right|}{\Vert \mathbf{w}\Vert }=\left|\delta \left(\mathbf{x},{\mathcal{H}}_{\mathbf{w},b}\right)\right|\text{\hspace{1em}(8.7)}$$

It is easy to see that the signed distance from the hyperplane is positive in half the space and negative in the other half. We put $H_{w,b}$The margin of is defined as close to $H_{w,b}$A collection of points. More precisely, let

$$\begin{array}{r}{\mathcal{H}}_{+}:=\left\{\mathbf{x}\in {\mathbb{R}}^p\mid {\mathbf{w}}^{\top }\mathbf{x}-b=1\right\}\\ H_{-}:=\left\{x\in R^p|w^{\top }x-b=-1\right\}\end{array}\text{\hspace{1em}(8.8;8.9)}$$

are two parallel to $H_{\mathbf{w},b}$affine hyperplane. Then $H_{w,b}$The margin of is defined as ${\mathcal{H}}_{+}\cup H_{-}$The convex hull of .

Using Lemma 8.1, we can see the thickness of this margin, which is $H_{+}$and $H_{-}$The distance between them is given by $\frac{2}{|\mathbf{w}\Vert }$given. Therefore, if we want to find an affine hyperplane $H_{\mathbf{w},b}$, allows for the "best" separation of two classes of data points. If we quantify "best" by allowing a maximum margin while still preventing data points from falling within this margin, we will have to maximize the 2 ∥ w ∥ \ $\frac{2}{\Vert \mathbf{w}\Vert }$, which brings us to linear SVM.

8.2 Basic Linear SVM

As mentioned above, SVM is a supervised learning method, so we start from $Starting\; from\; N$ -labeled training data,( xi , yi ) ∈ R p × { − 1 , 1 } , i = 1 , … , N \left(\mathbf{x}_{i}, y_{i}\right) \in \mathbb{R}^{p} \times\{-1,1\}, i=1, \ldots, N(xi​,yi​)∈Rp×{ −1,1},i=1,…,N. _ 这り，$y_i$is $1$ or$-1$ , indicating which of the two categories the data belongs to. For linear SVM, we must assume that this data is indeed linearly separable, that is, we must assume that there is an affine hyperplane${\mathcal{H}}_{\mathrm{w},b}$. There are no data points located at $H_{w,b}$Within the margin of , this constraint is equivalent to requiring that all belonging to $y_i=$The point of$1$$H_{+}$The distance of is positive, and all belonging to $y_i=-1$ data and${\mathcal{H}}_{-}$The distance is negative, that is.

$$\begin{array}{ll}{\mathbf{w}}^{\top }{\mathbf{x}}_i-b\ge +1& \text{ for }{y}_i=+1\\ w^{\top }{x}_i-b\le -1& \text{ for }{y}_i=-1\end{array}\text{\hspace{1em}(8.10)}$$

This can be written more concisely as
$$y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)\ge 1\text{ for all }i=1,\dots ,N\text{\hspace{1em}(8.11)}$$

Therefore, the task of finding the affine hyperplane that allows the maximum margin, while still separating the two classes, is described by the optimization problem
$$\max \frac{2}{\Vert \mathbf{w}{\Vert }^2}\text{ s.t. }{y}_i\left(w^{\top }{x}_i-b\right)\ge 1\text{ for all }i=1,\dots ,N\text{\hspace{1em}(8.12)}$$

或者等价地，以
$$\min \frac{1}{2}\Vert \mathbf{w}{\Vert }^2\text{ s.t. }{y}_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)\ge 1\text{ for all }i=1,\dots ,N\text{\hspace{1em}(8.13)}$$

In order to solve this restricted optimization problem, we need to introduce optimization conditions for these types of problems.

8.2.1 Karush-Kuhn-Tucker condition

We refer to the textbook Numerical Optimization by J. Nocedal and SJ Wright, 2nd edition, Springer 2006, for a more detailed insight into the topic of optimization. Consider the general problem
$$\begin{array}{rl}{\min }_{\mathbf{z}\in {\mathbb{R}}^n}& f(\mathbf{z})\\ \text{ s.t. }& c_i(\mathbf{z})=0\text{ for }i\in \mathcal{E}\\ & \text{ and }{c}_j\left(z\right)\ge 0\text{ for }j\in \mathcal{I}\end{array}\text{\hspace{1em}(8.14)}$$

and smooth real-valued functions $f,c_i$. Here $E$ represents equality constraint,$I$ stands for inequality constraint. For a certain point z that satisfies the constraints, we define its activity set as$\mathcal{A}(\mathbf{z})=E.$ _ In other words,$A\left(z\right)$ isz \mathbf{z}The index of all constraints for which$z\; exactly\; satisfies\; the\; equality\; condition.$

The Lagrangian function of the optimization problem (8.15) is given by
$$L(\mathbf{z},l)=f(\mathbf{z})-\sum _{i\in \mathcal{I}\cup \mathcal{E}}{l}_i{c}_i(\mathbf{z})\text{\hspace{1em}(8.15)}$$

Theorem 8.2 Karush-Kuhn-Tucker (KKT) conditions

任${\mathbf{z}}^{\star }$ is the solution of (8.15). In constraint function${}^{Under\; certain\; conditions\; of\; 2}$ , there exists a Lagrange multiplier$l^{\star }$ , making

$$\begin{array}{rl}{\nabla }_{\mathbf{z}}L\left({\mathbf{z}}^{\star },l^{\star }\right)& =0\\ c_i\left(z^{\star }\right)& =0\text{ for }i\in \mathcal{E}\\ c_i\left(z^{\star }\right)& \ge 0\text{ for }i\in \mathcal{I}\\ l_i^{\star }& \ge 0\text{ for }i\in I\\ l_i^{\star }{c}_i\left(z^{\star }\right)& =0\text{ for }i\in I\cup \mathcal{E}\end{array}\text{\hspace{1em}(8.16-8.21)}$$

Since the SVM optimization problem is convex (a convex objective function and constraints that define a convex feasible region), the KKT conditions are z ⋆ , λ ⋆ \ ${\mathbf{z}}^{\star },l^{\star Necessary}$ and sufficient conditions for being a solution. The last condition means that either constraint$i$ is in the active set, or the$The\; i$ component is zero, or possibly both.

${}^2$ This is satisfied if they are linear, so specifically for the case of linear SVM.

8.2.2 Lagrangian Duality

As before, we will consider the general optimization problem

$$\min f(\mathbf{z})\text{ s.t. }{c}_i(\mathbf{z})=0,i\in \mathcal{E},c_j\left(z\right)\ge 0,j\in \mathcal{I}$$

This is also known as the original problem. The corresponding Lagrangian function is defined as
$$L(\mathbf{z},l)=f\left(z\right)-\sum _{i\in \mathcal{I}\cup \mathcal{E}}{l}_i{c}_i\left(z\right)$$

With Lagrange multipliers (also called dual variables) $l_i\ge 0$ . Based on the Lagrangian function, we can create a new function for the objective function$f$ provides a lower bound. Since the dual variables are both positive numbers, that is,$l_i\ge 0$ , we know that$f(\mathbf{z})\ge$ $L(\mathbf{z},\lambda )$for all feasible$z$ . This leads to the definition of the Lagrangian dual function as

$$g\left(\lambda \right)=\underset{\mathbf{z}}{\inf }L(z,l)=\underset{z}{\inf }\left(f(\mathbf{z})-\sum _{i\in \mathcal{I}\cup \mathcal{E}}{\lambda }_i{c}_i(\mathbf{z})\right)\text{\hspace{1em}(8.22)}$$

对偶函数$g(\cdot )$是凹的，即使原来的问题不是凸的，因为它是仿射函数的逐点下确界（point-wise infimum of affine functions）。对偶形式产生了目标函数$f$的最优值$p^{\star }$的下限。对于任何$\mathbf{\lambda }\ge 0$，我们有$g(\mathbf{\lambda })\le p^{\star }$。

拉格朗日对偶问题是最大化问题

$$\underset{l}{\max }g\left(\lambda \right)\text{ s.t. }{l}_i\ge 0$$

Under certain conditions (which holds in the case of SVM), the minimum of the original problem coincides with the maximum of the dual problem, i.e. $d^{\star }=\max g\left(\lambda \right)=$ ${\inf }_{\mathbf{z}}f(\mathbf{z})=p^{\star }$ . This is called strong duality.

Remark
duality enables us to use convex optimization to calculate a lower bound on the optimal value of any problem, convex or not. However, the dual function $g$ may not be easy to calculate because it is itself defined as an optimization problem. whengg_Duality works best when$g\; can\; be\; written\; in\; closed\; form.$Even so, it may not be easy to find a solution to the dual problem, since not all convex problems are easy to solve.

8.2.3 Linear SVM: Primal and Dual Problem (Linear SVM: Primal and Dual Problem)

In the case of an infinitesimal range, it is possible to set a range of
$$\begin{array}{c}\underset{\mathbf{w},b,\mathbf{\lambda }\ge 0}{\min }L(\mathbf{w},b,l)\\ \text{ with }L(\mathbf{w},b,l)=\frac{1}{2}\Vert \mathbf{w}{\Vert }^2-\sum _i{l}_i\left(y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)-1\right)\\ =\frac{1}{2}\Vert w{\Vert }^2-\sum _i{l}_i{y}_i\left(w^{\top }{x}_i-b\right)+\sum _i{l}_i\end{array}\text{\hspace{1em}(8.23-25)}$$

This problem is strictly convex, so if it exists, its solution is unique. The gradient and optimization parameters of the Lagrangian function ( $w,b$ ) The relationship is

$${\nabla }_{(\mathbf{w},b)}L(\mathbf{w},b,l)=\left[\begin{array}{c}\mathbf{w}-{\sum }_i{l}_i{y}_i{\mathbf{x}}_i\\ {\sum }_i{l}_i{y}_i\end{array}\right]\text{\hspace{1em}(8.26)}$$

Therefore, the KKT condition $\left(8.17\right)$ and$(\; 8.21$ ) ($8.21$$)$$\left(8.21\right)$ get$\_$$\_$$\_$

$$\begin{array}{r}{\mathbf{w}}^{\star }-{\sum }_i{l}_i^{\star }{y}_i{\mathbf{x}}_i=0\\ {\sum }_i{l}_i^{\star }{y}_i=0\\ l_i^{\star }\left(y_i\left({\left({\mathbf{w}}^{\star }\right)}^{\top }{x}_i-b^{\star }\right)-1\right)=0.\end{array}\text{\hspace{1em}(8.27-29)}$$

This means that the solution $w^{\star }$ is a linear combination of points touching the boundary hyperplane, i.e. those located at${\mathcal{H}}_{+}$and ${\mathcal{H}}_{-}$center point. These points are nominally support points or vectors.

Now we can use these equations to derive a simple method to find the required optimal hyperplane parameters ${\mathbf{w}}^{\star }$ and$b^{\star }$ . First,$(8.27)$ and (8.28) are substituted into (8.25), and we get (omitted to improve readability${}^{\star }$ ) equation

$$\begin{array}{rl}& \frac{1}{2}\Vert \mathbf{w}{\Vert }^2-\sum _i{l}_i{y}_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)+\sum _i{l}_i\\ =& \frac{1}{2}\sum _{i,j}{l}_i{l}_j{y}_i{y}_j{\mathbf{x}}_i^{\top }{x}_j-\sum _{i,j}{l}_i{l}_j{y}_i{y}_j{x}_i^{\top }{x}_j+\sum _i{l}_i{y}_{i}b+\sum _i{l}_i\\ =& \sum _i{l}_i-\frac{1}{2}\sum _{i,j}{l}_i{l}_j{y}_i{y}_j{x}_i^{\top }{x}_j.\end{array}$$

This only depends on $The\; new\; formulation\; of\; \lambda$ is the dual form of the problem (see$(8.22)$ ) Let
$$L_D\left(l\right)=\sum _i{l}_i-\frac{1}{2}{l}^{\top }\mathbf{H}\mathbf{\lambda }\text{ s.t. }{l}_i\ge 0,\sum _i{l}_i{y}_i=0$$

where H \mathbf{H}The term of$H$$h_{ij}=y_i{y}_j{\mathbf{x}}_i^{\top }{x}_j$. The optimal Lagrange multiplier is found by solving the maximization problem.

$$\underset{l}{\max }\left(\sum _i{l}_i-\frac{1}{2}{l}^{\top }\mathbf{H}\mathbf{\lambda }\right)\text{ s.t. }{l}_i\ge 0,\sum _i{l}_i{y}_i=0\text{\hspace{1em}(8.30)}$$

This is a convex quadratic optimization problem that can be solved with a quadratic program (QP) solver (such as the function quadprog in Matlab). By plugging it into the formula $(8.27)$ , get$b^{\star }$ , and then through the formula$(8.29)$ Get${\mathbf{w}}^{\star }$ . Note that the Lagrange multiplier$l_i$Corresponds to point ${\mathbf{x}}_i$. result $l_i$is not equal to zero, then equation $(8.29)$意味着$y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)=1$ , that is, the corresponding${\mathbf{x}}_i$is ${\mathcal{H}}_{+}$or $H_{-}$of an element.

8.3 Soft Margin Linear SVM

Obviously, if the training set is not linearly separable, the above method will fail, because in this case, there are no points that satisfy the constraints. To overcome this obvious shortcoming, soft-margin $SVMallowsfor$ some misallocated data samples. It finds a compromise between large margin and degree of misclassification. This misclassification is performed by a set ofNNAdditional variables of$N$$X_i$To quantify, these variables are assumed to be nonnegative, resulting in constraints

$$y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)\ge 1-X_i\text{ for all }i=1,\dots ,N\text{\hspace{1em}(8.31)}$$

Therefore, in order to obtain a large margin while maintaining moderate misclassification, we consider the optimization problem

$$\begin{array}{ll}{\min }_{w,b,}& \frac{1}{2}\Vert \mathbf{w}{\Vert }_2^2+c{\sum }_{i=1}^N{X}_i\\ \text{ s.t. }& y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)\ge 1-X_i\text{ and }{X}_i\ge 0\forall i\end{array}\text{\hspace{1em}(8.32-33)}$$

Therefore, in order to obtain a large margin, at the same time the free parameter $c>0$ trades off between large margin and misclassification. Selected$The\; larger\; c$ is, the greater the penalty for violating the separation rule. As mentioned before, this is a quadratic programming problem and can be solved with a QP solver. The corresponding KKT conditions are discussed below. The Lagrangian of our soft-margin SVM is: To maintain moderate misclassification, we consider the optimization problem
$$L(\mathbf{w},b,x,l,m)=\frac{1}{2}\Vert \mathbf{w}{\Vert }^2+c\sum _i{X}_i-\sum _i{l}_i\left(y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)-1+X_i\right)-\sum _i{m}_i{X}_i\text{\hspace{1em}(8.34)}$$

Among them $m_i$is to enforce $X_i$Determine the Lagrange equation for the KKT function
$$\begin{array}{r}{\nabla }_{\mathbf{w}}L=\mathbf{w}-{\sum }_i{l}_i{y}_i{\mathbf{x}}_i=0\\ {\nabla }_{b}L={\sum }_i{l}_i{y}_i=0\\ {\nabla }_{X_i}L=c-l_i-m_i=0\\ y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)-1+X_i\ge 0\\ X_i\ge 0\\ l_i\ge 0\\ m_i\ge 0\\ l_i\left(y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)-1+X_i\right)=0\\ m_i{X}_i=0\end{array}\text{\hspace{1em}(8.35-43)}$$

Although a QP solver can already find the solution to this problem, we will now derive the corresponding dual form, since its appearance is very similar to the separable problem. Additionally, in order to extend SVM to work with kernels, the dual form is required. First, note that equation $(8.37)$ means$m_i=c-l_i.Insert$ it into equation$(8.34)$ , we can already eliminate$m_i$dependence.

$$\frac{1}{2}{\mathbf{w}}^{\top }\mathbf{w}-\sum _i{l}_i\left(y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)-1\right)\text{\hspace{1em}(8.44)}$$

Next, we use $(8.35)$的事实：$w={\sum }_i{l}_i{y}_i{\mathbf{x}}_i$and substitute it into (8.44). By using attributes $\left(8.36\right)$ , we get the dual form LD (λ) = ∑ i λ i − 1 2 λ ⊤ H λ (8.45) L_{D}(\boldsymbol{\lambda})=\sum_{i} \ $lambda\_$
$$L_D\left(l\right)=\sum _i{l}_i-\frac{1}{2}{l}^{\top }\mathbf{H}\mathbf{\lambda }\text{\hspace{1em}(8.45)}$$

Matrix $\mathbf{H}$的项 $h_{ij}=y_i{y}_j{\mathbf{x}}_i^{\top }{x}_j$ 在$0\le l_i\le c$ 与 ${\sum }_i{l}_i{y}_i=$under the constraints of$0\; .$Therefore, the dual problem has the form

$$\underset{l}{\max }{L}_D\left(l\right)\text{ s.t. }0\le l_i\le c,\sum _i{l}_i{y}_i=0\text{\hspace{1em}(8.46)}$$

After that, the solution is $\mathbf{w}={\sum }_i{l}_i{y}_i{\mathbf{x}}_i$given. Therefore, the only difference from the separable case is that $l_i$There is an upper $c$ . Equation$\left(8.37\right)$ and$(\; 8.43\; )$$($$8.43$$(8.43)$ Semantic outcome$l_i<c，x_i=0$ . Therefore, any training sample${\mathbf{x}}_i$（如果$0<l_i<c$ ) is a support vector. Furthermore, for any support vector$x_i$Equation $(8.42)$简化为$y_i\left({\mathbf{w}}^{\top }{\mathbf{x}}_i-b\right)+1=0$ , can be used to calculate$b^{\star }$ . In order to obtain a more stable solution, it is recommended to use the average of all points in the support zone. Specifically, it is defined as

$$b^{\star }=\frac{1}{N_{Supp}}\sum _{i\in Supp}\left({\left({\mathbf{w}}^{\star }\right)}^{\top }{\mathbf{x}}_i-y_i\right)\text{\hspace{1em}(8.47)}$$

where $Supp$ represents the support index,$N_{Supp}$Represents the number of support vectors.

8.4 Nuclear SVM

As we saw before, for a set of training samples ( xi , yi ) , i = 1 , … , N , xi ∈ R p , yi ∈ { − 1 , 1 } \left(\mathbf{x}_{i }, y_{i}\right), i=1, \ldots, N, \mathbf{x}_{i} \in \mathbb{R}^{p}, y_{i} \in\{-1 ,1\}(xi​,yi​),i=1,…,N,xi​∈Rp,yi​∈{ −1,1 } can be rewritten as dual form

$$\begin{array}{ll}{\max }_{}& {\sum }_i{l}_i-\frac{1}{2}{l}^{\top }\mathbf{H}\mathbf{\lambda }\\ \text{ s.t. }& 0\le l_i\le c,{\sum }_i{l}_i{y}_i=0\end{array}\text{\hspace{1em}(8.48)}$$

With Gram matrix $H$，其项定义为$h_{ij}=y_i{y}_j{\mathbf{x}}_i^{\top }{x}_j$. We talked about the kernel trick in the chapter on kernel PCA, i.e. using the nonlinear function $\varphi maps$ the training samples to the high-dimensional Hilbert space$H$ , whose inner product is$⟨\cdot ,\cdot {⟩}_{\mathcal{H}}$And directly use the kernel function $\kappa$ expression inner product. In other words, the kernel function$\kappa interpretation$ ( x , y ) = ⟨ ϕ ( x ) , ϕ ( y ) ⟩ H \kappa(\mathbf{x}, \mathbf{y})=\langle\phi(\mathbf{x}),\$k(x,\mathbf{y})=⟨\varphi \left(x\right),\varphi \left(y\right){⟩}_{\mathcal{H}}$Map two points to $R.$ _ The same is true in this case. Inner product${\mathbf{x}}^{\kappa \; (\; x},\; y$) = ( α x ⊤ y + β ) γ polynomial kernel κ ( x , y ) = exp ⁡ ( − ∥ x − y∥
$$\begin{array}{ll}k(x,\mathbf{y})={\left(\alpha {\mathbf{x}}^{\top }\mathbf{y}+b\right)}^c& \text{ polynomial kernel }\\ k(x,\mathbf{y})=\exp \left(-\Vert \mathbf{x}-\mathbf{y}{\Vert }^2/\left(2p^2\right)\right)& \text{ radial basis function kernel }\\ k(x,y)=\mathrm{fishy}(\gamma {\mathbf{x}}^{\top }\mathbf{y}-d& \text{ sigmoid kernel }\end{array}\text{\hspace{1em}(8.49-51)}$$

且$\mathrm{fishy}\left(x\right)=\left(e^x-e^{-x}\right)/\left(e^x+e^{-x}\right)$. One of these kernel functions is then used to define the terms of the Gram matrix by

$$h_{ij}=y_i{y}_{j}K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)\text{\hspace{1em}(8.52)}$$

A few notes on the aforementioned cores.

• In the case of polynomial kernel, when the original signal is in ${\mathbb{R}}^{}$When${}^p$ , Hilbert space H \mathcal{H}DimensionH \mathcal{H}$of\; H$$\mathcal{H}$为$\left(\begin{array}{c}p+d\\ d\end{array}\right)$。

• The radial basis function kernel is also often called a Gaussian kernel, which describes a nonlinear function $\varphi$ , which maps to an infinite-dimensional Hilbert space$H.$。

• The sigmoid kernel is only in $c,$Under a specific choice of$\delta$$\Vert \mathbf{x}{\Vert }^{2.}$ Generate an spd kernel matrix under specific conditions$.$

Solve the following problems

$$\begin{array}{ll}{\max }_{}& {\sum }_i{l}_i-\frac{1}{2}{l}^{\top }\mathbf{H}\mathbf{\lambda }\\ \text{ s.t. }& 0\le l_i\le c,{\sum }_i{l}_i{y}_i=0\end{array}\text{\hspace{1em}(8.53)}$$

其中$h_{ij}=y_i{y}_{j}K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)$ provides the Lagrange coefficient$l^{\star }$ , but how do we use this result to classify new points? First, note that according to the classification rule in the linear case (that is, use$\mathrm{sign}\left({\mathbf{w}}^{\top }\mathbf{z}-b\right)$ Rule pair$z$ to classify, we can express the decision function in the following way

$$f(\mathbf{z})=\sum _{i\in Supp}{l}_i{y}_{i}K\left({\mathbf{x}}_i,\mathbf{z}\right)-b\text{\hspace{1em}(8.54)}$$

where $Supp$ represents support (i.e., for all$i$有 $0<l_i\le c$ ). We use$The\; sign\; of\; f\left(z\right)$ is used to assign labels.

The only remaining component we need is the factor $b$ . From the KKT conditions of the original problem, we know the equation

$$y_i\left({⟨\varphi \left({\mathbf{x}}_i\right),\mathbf{w}⟩}_{\mathcal{H}}\right)-b)-1=0$$

对 Supp ⁡ \operatorname{Supp} any iiin$S$$u$$p$$p$$i$ must be established. Vector$w$ is a high-dimensional Hilbert space$An\; element\; of\; H$ can be rewritten as the sum of the mapped support vectors${\sum }_{j\in \mathrm{Supp}}{l}_i{y}_i\varphi \left({\mathbf{x}}_i\right)$ . Plugging this into the previous equation we get

$$b=\sum _{j\in Supp}\left(l_j{y}_j{⟨\varphi \left(x_i\right),\varphi \left(x_j\right)⟩}_{\mathcal{H}}\right)-y_i$$

For any ii in support$For\; i$ . To make this result more robust,$i$进行平均，得到
$$b=\frac{1}{N_{Supp}}\sum _{i\in Supp}\left(\sum _{j\in Supp}\left(l_j{y}_j{⟨\varphi \left({\mathbf{x}}_i\right),\varphi \left(x_j\right)⟩}_{\mathcal{H}}\right)-y_i\right)$$

Among them $N_{Supp}$is the number of support vectors. Therefore, by replacing the inner product with the kernel function, we can calculate $b$为
$$b=\frac{1}{N_{Supp}}\sum _{i\in Supp}\left(\sum _{j\in Supp}\left(l_j{y}_{j}K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)\right)-y_i\right)\text{\hspace{1em}(8.55)}$$