[Loss function] SmoothL1Loss Smooth L1 loss function

1 Introduction

torch.nn.SmoothL1Loss is a loss function in PyTorch, usually used for regression problems. It is a combination of L1 loss and L2 loss, aiming to reduce sensitivity to outliers.

loss_function = nn.SmoothL1Loss(reduction='mean', beta=1.0)

2. Parameters

1. size_average (deprecated) : Previously used to determine whether the loss should be averaged per element. If set to  False, losses are summed. This option is now deprecated and  reduction parameters should be used instead.

2. reduce (deprecated) : This is also an old parameter that specifies whether to apply reduction. Now also  reduction replaced by parameters.

3. reduction : Specifies the reduction method to be applied to the output. Optional values ​​are:

   • 'none': No reduction is applied.
   • 'mean': Calculates the average of losses.
   • 'sum': Calculates the sum of losses.

4. beta : Used to determine the smooth transition point. For errors less  beta than , the loss function becomes L2 loss, and for  beta errors greater than , it becomes L1 loss.

3. Image

        In the image of the Smooth L1 loss function, when the difference between the predicted value and the true value is small (less than  beta, which defaults to 1.0 here), it is calculated similarly to the L2 loss (squared error). When the difference is large, it is calculated similar to L1 loss (absolute error). This hybrid property makes the Smooth L1 loss less sensitive to outliers and more stable in optimization. ​ 

4. Examples

Suppose we have the following situation: we are training a model to predict some continuous value, such as house prices. We have the following target values ​​(true values) and predicted values:

• Target (real value): [1.5, 2.0, 3.0]
• predict: [1.4, 2.1, 2.9]

As the loss function we use  SmoothL1Loss :

import torch
import torch.nn as nn

# 定义目标和预测值
targets = torch.tensor([1.5, 2.0, 3.0])
predictions = torch.tensor([1.4, 2.1, 2.9])

# 创建 SmoothL1Loss 实例
loss_function = nn.SmoothL1Loss(reduction='mean', beta=1.0)

# 计算损失
loss = loss_function(predictions, targets)
print(loss)

        In this example, the loss function will calculate the Smooth L1 loss between the target and prediction and return its average. If the difference between prediction and target is less than  beta(1.0 in this case), then it applies the squared form of the L2 loss; if the difference is greater  beta, it applies the absolute form of the L1 loss. This blending makes the Smooth L1 loss less sensitive to outliers, especially when the predicted values ​​differ greatly from the true values.

5, reference

【pytorch】nn.SmoothL1Loss function usage_nn.smoothl1loss()-CSDN Blog

PyTorch study notes: nn.SmoothL1Loss——Smooth L1 loss_nn.smoothl1loss()-CSDN Blog