1. NMS
1.1. NMS概述
非极大值抑制（Non-Maximum Suppression， NMS），顾名思义就是抑制不是极大值的元素，用于目标检测中，就是提取置信度高的目标检测框，而抑制置信度低的误检框。一般来说，用在当解析模型输出到目标框时，目标框会非常多，具体数量由anchor数量决定，其中有很多重复的框定位到同一个目标，NMS用来去除这些重复的框，获得真正的目标框。
如上图所示，人、马、车都有很多检测框，通过NMS，得到唯一的检测框。
1.2. NMS流程
依靠分类器得到多个候选框，以及关于候选框中属于类别的概率值，根据分类器得到的类别分类概率做排序，具体算法流程如下：
（1）将bounding box按照confidence从高到低排序，并记录当前confidence最大的bounding box。
（2）计算最大confidence对应的bounding box与剩下所有bounding box的IoU，移除所有大于IoU阈值的bounding box。（为什么要删除，是因为你超过设定阈值，认为两个框是在检测同一个物体）
（3）对剩下的bounding box循环执行（2）和（3），直到所有的bounding box满足要求（即不再移除bounding box）
1.3. NMS代码实现
import math
import numpy as np
def iou(box1, box2):
# box format: xyxy
area1 = (box1[3] - box1[1]) * (box1[2] - box1[0])
area2 = (box2[3] - box2[1]) * (box2[2] - box2[0])
inter_area = (min(box1[2], box2[2]) - max(box1[0], box2[0])) * \
(min(box1[3], box2[3]) - max(box1[1], box2[1]))
return inter_area / area1 + area2 - inter_area
def spm(iou, mode='linear', sigma=0.3):
# score penalty mechanism (soft-nms)
if mode == 'linear':
return 1 - iou
elif mode == 'gaussian':
return math.e ** (- (iou ** 2) / sigma)
else:
raise NotImplementedError
def NMS(lists, conf_thre, iou_thre, soft=True, soft_thre=0.001):
# Non-Maximum Suppression
lists = filter(lambda x: x[4] >= conf_thre, lists)
lists = sorted(lists, key=lambda x: x[4], reverse=True)
keep = []
while lists:
m = lists.pop(0)
keep.append(m)
for i, pred in enumerate(lists):
_iou = iou(m, pred)
if _iou >= iou_thre:
if soft:
pred[4] *= spm(_iou, mode='gaussian', sigma=0.3)
keep.append(lists.pop(i))
else:
lists.pop(i)
if soft:
keep = list(filter(lambda x: x[4] >= soft_thre, keep))
keep = sorted(keep, key=lambda x: x[4], reverse=True)
return keep
if __name__ == '__main__':
np.random.seed(0)
x1y1 = np.random.randint(0, 300, (300, 2)) / 600 # assume image shape is (600, 600)
x2y2 = np.random.randint(300, 600, (300, 2)) / 600 # pixel to normalized
boxes = np.concatenate((x1y1, x2y2), 1)
scores = np.random.rand(300, 1)
lists = list(np.concatenate((boxes, scores), 1))
detections = NMS(lists, conf_thre=0.1, iou_thre=0.7, soft=False, soft_thre=0.1)
print(len(detections), detections)
2. soft-NMS
2.1 soft-NMS概述
传统的NMS算法首先在被监测图片中产生一系列的检测框B以及对应的分数S。当选中最大分数的检测框M时，该框从集合B中移出并放入最终检测结果集合D。与此同时，集合B中任何与检测框M重叠部分大于一定阈值的检测框也将随之移除。但是传统的NMS存在一定的问题：如果一个物体在另一个物体重叠区域出现，即当两个目标框接近时，分数更低的框就会因为与之重叠面积过大而被删掉，从而导致对该物体的检测失败并降低了算法的平均检测率。
如上图所示，检测算法本来应该输出两个检测框，但是传统的NMS由于绿框的得分较低且绿框和红框的IoU大于设定的阈值，因此会被过滤掉，导致只检测出一匹马，显然这样的算法设计是不合理的。NMS直接粗暴的将和得分最大的bbox的IoU大于阈值的bbox得分置0。那么有没有缓和（soft）一点的方式，这就引出了soft-NMS，简而言之，soft-NMS就是用一个稍微小一点的分数替代原有的分数，而非直接粗暴的置0。
2.2 soft-NMS流程
soft-NMS算法流程如下图所示：
传统的NMS，当前检测框和最高分检测框的IoU大于阈值时，直接将该检测框的得分置0，其算法如上图红色框所示，这将导致重叠区域较大的目标框被漏检。NMS算法可以用下面的式子表示：（其中s_i表示当前检测框的得分，N_t为IoU的阈值，M为得分最高的检测框。）
为了改变NMS这种hard threshold做法，并遵循iou越大，得分越低的原则（iou越大，越有可能是false positiive），就可以用下面的公式来表示soft NMS：
但是上面这个公式是不连续的，这样会导致bbox集合中的score出现断层，因此就有了下面这个soft NMS式子（也是大部分实验中采用的式子），将当前检测框得分乘以一个权重函数，该函数会衰减与最高得分检测框M有重叠的相邻检测框的分数，越是与M框高度重叠的检测框，其得分衰减越严重，为此选择高斯函数为权重函数，从而修改其删除检测框的规则。高斯权重函数如下所示（δ通常取0.3）。
2.3 soft-NMS代码实现
import numpy as np
cimport numpy as np
cdef inline np.float32_t max(np.float32_t a, np.float32_t b):
return a if a >= b else b
cdef inline np.float32_t min(np.float32_t a, np.float32_t b):
return a if a <= b else b
def cpu_soft_nms(np.ndarray[float, ndim=2] boxes, float sigma=0.5, float Nt=0.3, float threshold=0.001, unsigned int method=0):
cdef unsigned int N = boxes.shape[0]
cdef float iw, ih, box_area
cdef float ua
cdef int pos = 0
cdef float maxscore = 0
cdef int maxpos = 0
cdef float x1,x2,y1,y2,tx1,tx2,ty1,ty2,ts,area,weight,ov
for i in range(N):
maxscore = boxes[i, 4]
maxpos = i
tx1 = boxes[i,0]
ty1 = boxes[i,1]
tx2 = boxes[i,2]
ty2 = boxes[i,3]
ts = boxes[i,4]
pos = i + 1
# get max box
while pos < N:
if maxscore < boxes[pos, 4]:
maxscore = boxes[pos, 4]
maxpos = pos
pos = pos + 1
# add max box as a detection
boxes[i,0] = boxes[maxpos,0]
boxes[i,1] = boxes[maxpos,1]
boxes[i,2] = boxes[maxpos,2]
boxes[i,3] = boxes[maxpos,3]
boxes[i,4] = boxes[maxpos,4]
# swap ith box with position of max box
boxes[maxpos,0] = tx1
boxes[maxpos,1] = ty1
boxes[maxpos,2] = tx2
boxes[maxpos,3] = ty2
boxes[maxpos,4] = ts
tx1 = boxes[i,0]
ty1 = boxes[i,1]
tx2 = boxes[i,2]
ty2 = boxes[i,3]
ts = boxes[i,4]
pos = i + 1
# NMS iterations, note that N changes if detection boxes fall below threshold
while pos < N:
x1 = boxes[pos, 0]
y1 = boxes[pos, 1]
x2 = boxes[pos, 2]
y2 = boxes[pos, 3]
s = boxes[pos, 4]
area = (x2 - x1 + 1) * (y2 - y1 + 1)
iw = (min(tx2, x2) - max(tx1, x1) + 1)
if iw > 0:
ih = (min(ty2, y2) - max(ty1, y1) + 1)
if ih > 0:
ua = float((tx2 - tx1 + 1) * (ty2 - ty1 + 1) + area - iw * ih)
ov = iw * ih / ua #iou between max box and detection box
if method == 1: # linear
if ov > Nt:
weight = 1 - ov
else:
weight = 1
elif method == 2: # gaussian
weight = np.exp(-(ov * ov)/sigma)
else: # original NMS
if ov > Nt:
weight = 0
else:
weight = 1
boxes[pos, 4] = weight*boxes[pos, 4]
# if box score falls below threshold, discard the box by swapping with last box
# update N
if boxes[pos, 4] < threshold:
boxes[pos,0] = boxes[N-1, 0]
boxes[pos,1] = boxes[N-1, 1]
boxes[pos,2] = boxes[N-1, 2]
boxes[pos,3] = boxes[N-1, 3]
boxes[pos,4] = boxes[N-1, 4]
N = N - 1
pos = pos - 1
pos = pos + 1
keep = [i for i in range(N)]
return keep
def cpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh):
cdef np.ndarray[np.float32_t, ndim=1] x1 = dets[:, 0]
cdef np.ndarray[np.float32_t, ndim=1] y1 = dets[:, 1]
cdef np.ndarray[np.float32_t, ndim=1] x2 = dets[:, 2]
cdef np.ndarray[np.float32_t, ndim=1] y2 = dets[:, 3]
cdef np.ndarray[np.float32_t, ndim=1] scores = dets[:, 4]
cdef np.ndarray[np.float32_t, ndim=1] areas = (x2 - x1 + 1) * (y2 - y1 + 1)
cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]
cdef int ndets = dets.shape[0]
cdef np.ndarray[np.int_t, ndim=1] suppressed = \
np.zeros((ndets), dtype=np.int)
# nominal indices
cdef int _i, _j
# sorted indices
cdef int i, j
# temp variables for box i's (the box currently under consideration)
cdef np.float32_t ix1, iy1, ix2, iy2, iarea
# variables for computing overlap with box j (lower scoring box)
cdef np.float32_t xx1, yy1, xx2, yy2
cdef np.float32_t w, h
cdef np.float32_t inter, ovr
keep = []
for _i in range(ndets):
i = order[_i]
if suppressed[i] == 1:
continue
keep.append(i)
ix1 = x1[i]
iy1 = y1[i]
ix2 = x2[i]
iy2 = y2[i]
iarea = areas[i]
for _j in range(_i + 1, ndets):
j = order[_j]
if suppressed[j] == 1:
continue
xx1 = max(ix1, x1[j])
yy1 = max(iy1, y1[j])
xx2 = min(ix2, x2[j])
yy2 = min(iy2, y2[j])
w = max(0.0, xx2 - xx1 + 1)
h = max(0.0, yy2 - yy1 + 1)
inter = w * h
ovr = inter / (iarea + areas[j] - inter)
if ovr >= thresh:
suppressed[j] = 1
return keep