Evaluating model metrics optimization guide

All content in this article is compiled from Coursera - Advanced Machine Learning- How to Win a Data Science Competition: Learn from Top Kagglers

1. Define evaluation metrics

1. Basic Concepts

• In general, the optimization matrix is ​​defined by the organizer according to business needs
• In some cases, the optimization objective is not easy to quantify, and intuitive judgment is required to convert to other optimization matrices
• Observe whether the trend of optimization matrix optimization converges well, if not, take other optimization matrices

2. Basic model

reg regression

• MSE mean variance
  
• RMSE MSE (standard deviation) under the root sign: make the difference value and the object value in the same level, rather than being squared
  
• R-squared
  
• MAE mean absolute difference: it will not punish the maximum and minimum values ​​too much, and is more tolerant of outliers
  
• MSPE\MSAE relative error: When calculating the error, consider the volume of the data itself (999-1000, which is different from the error of 1-2)
  
• RMSLE log mean difference under the root sign
  

classi classification

• accuracy
  
• logloss: tends to tolerate small errors and penalize obvious ones
  
• AUC: Suitable for binary classification. Calculate the degree of order after classification; auc is pair-wise
  

def aucfun(act,pred):
    fpr, tpr, thresholds = sklearn.metrics.roc_curve(act, pred, pos_label=1)
    return metrics.auc(fpr, tpr)

• Cohens Kappa: Define the accuracy rate for the baseline to avoid meaningless high accuracy
  

# **计算loss**
def soft_kappa(preds, dtrain):
    '''
        Having predictions `preds` and targets `dtrain.get_label()` this function coumputes soft kappa loss.
        NOTE, that it assumes `mean(target) = 0`.

    '''
    target = dtrain.get_label()
    return 'kappa' ,  -2 * target.dot(preds) / (target.dot(target) + preds.dot(preds))

# **计算一阶二阶导数**
def soft_kappa_grad_hess(y, p):
    '''
        Returns first and second derivatives of the objective with respect to predictions `p`. 
        `y` is a vector of corresponding target labels.  
    '''
    norm = p.dot(p) + y.dot(y)

    grad = -2 * y / norm + 4 * p * np.dot(y, p) / (norm ** 2)
    hess = 8 * p * y / (norm ** 2) + 4 * np.dot(y, p) / (norm ** 2)  - (16 * p ** 2 * np.dot(y, p)) / (norm ** 3)
    return grad, hess

• Quadratic weighted
  

3. Model training optimization

• Directly optimized: MSE, Logloss
• It cannot be directly optimized. You need to train and optimize another model first, and then test the results on the final model: MSEPE, MAPE, RMSLE
• Optimize another model and get the final model after processing: Accuracy, Kappa

Second, the evaluation model in the model

1. Model and evaluation model

2. Decision tree

• Information Dysprosium
• gini coefficient: the more cluttered the categories contained in the population, the larger the GINI index
• Advantages and disadvantages:

1、Gini is intended for continuous attributes, and Entropy for attributes that occur in classes (e.g. colors)
2、“Gini” will tend to find the largest class, and “entropy” tends to find groups of classes that make up ~50% of the data((http://paginas.fe.up.pt/~ec/files_1011/week%2008%20-%20Decision%20Trees.pdf))
3、“Gini” to minimize misclassification
4、“Entropy” for exploratory analysis
5、Some studies show this doesn’t matter – these differ less than 2% of the time
6、Entropy may be a little slower to compute