Boosting algorithm, often seen and often new
Integrated learning: It is reflected in the ability to integrate some weak learners into a strong learner.
(1) Definition of weak learner vs strong learner:
Is the strong learner more accurate? Is that all?
The popular understanding of a strong learner is that the prediction accuracy is very high. Given enough training samples, it can eventually achieve an arbitrarily small error rate.
However, the prediction accuracy of the weak learner is very low, which may only be better than random guessing (1%), and there is a limit to the prediction accuracy, that is, no matter how much the number of samples is increased, the prediction accuracy cannot be improved.
In practice, it may be difficult to find a strong learner, but it is very simple to find a weak learner. The Boosting algorithm is able to integrate different (not required to be the same class, it can be neural network + decision tree) weak learners into a strong learner to achieve the goal of "three cobblers, better than Zhuge Liang".
(2) Boostings algorithm instance: Adaboost
The principle of Adaboost is to continuously iterate on the previous mistakes by adjusting the weight (increase) of the error sample and the weight of the weak learner in each round, and the final learner becomes a relatively strong learner.
Does Adaboost's advantages end there? That will not become such an algorithm written in textbooks. In fact, another advantage is that it can prevent overfitting very well. That is to say, after the model works well on the training set, the effect on the test set will not decrease, but can continue to improve.
How did you do it? In fact, it is reflected in the adoption of the "margin" idea. Recall that SVM maximizes the distance between the sample and the interface to achieve an optimal segmentation. The boosting paper mentioned that his loss function actually includes such a margin, so that after the error rate on the training data reaches zero, the effect can continue to improve and continuously increase the confidence in decision-making.
(3) What are the applications of the boosting algorithm?
For example: face image segmentation. Now the face can be detected in real time in the mobile phone camera, and whether a picture is a face is determined. The weak learner can simply check whether a certain value of the face feature is greater than a certain threshold, which can be simply described by a conditional judgment statement. Of course, the effect is not very good, but the boosting algorithm can be used to solve it very well.
In addition, XGboost has become a common award-winning algorithm in Kaggle competitions. It is an algorithm that implements Gradient Boosting, with fast calculation speed and low memory consumption. Author: Princess Mononoke https://www.bilibili.com/read/cv14581546?spm_id_from=333.999.0.0 Source: bilibili