Click on the top " AI proper way ", select the "star" Public No.
Heavy dry goods, the first time served
Li Hang "statistical learning methods" can be said to machine learning introductory book, many machine-learning courses, Internet companies interview, written questions, many of them refer to this book. Previously, the red stone on the public this number is also published some notes about the book and some of Python code, the purpose is for everyone to chew on this book bring some convenience. Just, Ph.D. red stone found in the Yellow Sea on its wide GitHub updated the "statistical learning methods" Python code, could not wait to share with everyone.
The reason is "a statistical learning method," the first edition was published in 2012, contains a number of major supervised learning algorithms and models. May 1, 2019, "statistical learning methods," the second edition of the official release, through the efforts of six years, based on the first edition adds the main algorithms and unsupervised learning model.
The second edition of the directory is:
Learning of supervision on a
Chapter 1 Introduction to statistical learning and supervised learning Chapter 2 Perceptron Chapter 3 k-nearest neighbor Chapter 4 Naive Bayesian Chapter 5, decision trees Chapter 6 logistic regression model with Optimum Chapter 7 SVM Chapter 8 upgrade method Chapter 9 EM algorithm and its promotion Chapter 10 hidden Markov model Chapter 11 CRFs Chapter 12 supervised learning methods are summarized first two unsupervised learning Introduction Chapter 13 unsupervised learning Chapter 14 clustering Chapter 15 Singular value decomposition Chapter 16 principal component analysis Chapter 17 latent semantic analysis latent semantic analysis probability Cap 18 Cap 19 Markov chain Monte Carlo
Chapter 20 potential Dirichlet allocation
Chapter 21 PageRank algorithm
Chapter 22 Unsupervised Learning Methods
Appendix A gradient descent
Appendix B Newton method and quasi-Newton method
Appendix C Lagrange duality
Appendix D substantially subspace matrix
Properties defined in Appendix E KL divergence and Dirichlet distribution of
For the new additions, Dr Hong Hai wide of the original source code GitHub be updated with new content, put direct address:
https://github.com/fengdu78/lihang-code
The revised some errors, increase an overview of each chapter, the update before completion of 12 chapters, and will increase the content of the second edition.
Modify major errors include:
max_count error Chapter 3 k nearest neighbor methods
Chapter 10 Hidden Markov Models viterbi index error
Addition:
Each chapter of the summary increase
Content project currently contains the following screenshot:
Currently, the project has been the star of the harvest 5000 +.
Python code
Below, support vector machine, for example, we can review the complete sample code of SVM:
class SVM:
def __init__(self, max_iter=100, kernel='linear'):
self.max_iter = max_iter
self._kernel = kernel
def init_args(self, features, labels):
self.m, self.n = features.shape
self.X = features
self.Y = labels
self.b = 0.0
# 将Ei保存在一个列表里
self.alpha = np.ones(self.m)
self.E = [self._E(i) for i in range(self.m)]
# 松弛变量
self.C = 1.0
def _KKT(self, i):
y_g = self._g(i) * self.Y[i]
if self.alpha[i] == 0:
return y_g >= 1
elif 0 < self.alpha[i] < self.C:
return y_g == 1
else:
return y_g <= 1
# g(x)预测值，输入xi（X[i]）
def _g(self, i):
r = self.b
for j in range(self.m):
r += self.alpha[j] * self.Y[j] * self.kernel(self.X[i], self.X[j])
return r
# 核函数
def kernel(self, x1, x2):
if self._kernel == 'linear':
return sum([x1[k] * x2[k] for k in range(self.n)])
elif self._kernel == 'poly':
return (sum([x1[k] * x2[k] for k in range(self.n)]) + 1)**2
return 0
# E（x）为g(x)对输入x的预测值和y的差
def _E(self, i):
return self._g(i) - self.Y[i]
def _init_alpha(self):
# 外层循环首先遍历所有满足0<a<C的样本点，检验是否满足KKT
index_list = [i for i in range(self.m) if 0 < self.alpha[i] < self.C]
# 否则遍历整个训练集
non_satisfy_list = [i for i in range(self.m) if i not in index_list]
index_list.extend(non_satisfy_list)
for i in index_list:
if self._KKT(i):
continue
E1 = self.E[i]
# 如果E2是+，选择最小的；如果E2是负的，选择最大的
if E1 >= 0:
j = min(range(self.m), key=lambda x: self.E[x])
else:
j = max(range(self.m), key=lambda x: self.E[x])
return i, j
def _compare(self, _alpha, L, H):
if _alpha > H:
return H
elif _alpha < L:
return L
else:
return _alpha
def fit(self, features, labels):
self.init_args(features, labels)
for t in range(self.max_iter):
# train
i1, i2 = self._init_alpha()
# 边界
if self.Y[i1] == self.Y[i2]:
L = max(0, self.alpha[i1] + self.alpha[i2] - self.C)
H = min(self.C, self.alpha[i1] + self.alpha[i2])
else:
L = max(0, self.alpha[i2] - self.alpha[i1])
H = min(self.C, self.C + self.alpha[i2] - self.alpha[i1])
E1 = self.E[i1]
E2 = self.E[i2]
# eta=K11+K22-2K12
eta = self.kernel(self.X[i1], self.X[i1]) + self.kernel(
self.X[i2],
self.X[i2]) - 2 * self.kernel(self.X[i1], self.X[i2])
if eta <= 0:
# print('eta <= 0')
continue
alpha2_new_unc = self.alpha[i2] + self.Y[i2] * (
E1 - E2) / eta #此处有修改，根据书上应该是E1 - E2，书上130-131页
alpha2_new = self._compare(alpha2_new_unc, L, H)
alpha1_new = self.alpha[i1] + self.Y[i1] * self.Y[i2] * (
self.alpha[i2] - alpha2_new)
b1_new = -E1 - self.Y[i1] * self.kernel(self.X[i1], self.X[i1]) * (
alpha1_new - self.alpha[i1]) - self.Y[i2] * self.kernel(
self.X[i2],
self.X[i1]) * (alpha2_new - self.alpha[i2]) + self.b
b2_new = -E2 - self.Y[i1] * self.kernel(self.X[i1], self.X[i2]) * (
alpha1_new - self.alpha[i1]) - self.Y[i2] * self.kernel(
self.X[i2],
self.X[i2]) * (alpha2_new - self.alpha[i2]) + self.b
if 0 < alpha1_new < self.C:
b_new = b1_new
elif 0 < alpha2_new < self.C:
b_new = b2_new
else:
# 选择中点
b_new = (b1_new + b2_new) / 2
# 更新参数
self.alpha[i1] = alpha1_new
self.alpha[i2] = alpha2_new
self.b = b_new
self.E[i1] = self._E(i1)
self.E[i2] = self._E(i2)
return 'train done!'
def predict(self, data):
r = self.b
for i in range(self.m):
r += self.alpha[i] * self.Y[i] * self.kernel(data, self.X[i])
return 1 if r > 0 else -1
def score(self, X_test, y_test):
right_count = 0
for i in range(len(X_test)):
result = self.predict(X_test[i])
if result == y_test[i]:
right_count += 1
return right_count / len(X_test)
def _weight(self):
# linear model
yx = self.Y.reshape(-1, 1) * self.X
self.w = np.dot(yx.T, self.alpha)
return self.w
In fact, I looked, the project not only includes sample code SVM, but also reading notes and summarize the corresponding summary.
"Statistical learning methods" Courseware
Author Chun Yuan: Tsinghua University, Shenzhen Graduate School, PPT courseware provides the first edition of the book's 12 chapters.
Courseware obtain the address:
Link: https: //pan.baidu.com/s/1_boHMIg6DqS7bgFuxlWF7Q extraction code: ffxy
Additional Resources
Summary compilation of relevant resources on Li Hang "statistical learning methods" before I had published, summarized below, see the article:
Li Hang "statistical learning" study notes
Li Hang latest resources "statistical learning methods," notes, Python code is readily available!
References:
https://github.com/wzyonggege/statistical-learning-method
https://github.com/WenDesi/lihang_book_algorithm
https://blog.csdn.net/tudaodiaozhale
