文章目录
提示：这里可以添加本文要记录的大概内容：
论文：
FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking**](http://arxiv.org/abs/2004.01888）
代码：https://github.com/ifzhang/FairMOT
0.安装步骤
1.首先创建虚拟环境，并安装相应包
conda create -n FairMOT
conda activate FairMOT
conda install pytorch==1.7.0 torchvision==0.8.0 cudatoolkit=10.2 -c pytorch
cd ${
FAIRMOT_ROOT}
pip install cython
pip install -r requirements.txt
2.编译DCNv2(可变形卷积)
这里必须用pytorch1.7.1或者1.7.0。因为作者提供的dcnv2编译代码是基于这两个torch版本的
克隆链接： https://github.com/ifzhang/DCNv2/tree/pytorch_1.7
git clone -b pytorch_1.7 https://github.com/ifzhang/DCNv2.git
cd DCNv2
./make.sh
3.安装ffmppeg(测试用)
ffmpeg主要用于编辑连续帧图片与视频，实现转换：
1.下载代码： http://ffmpeg.org/download.html
2.安装 yasm
sudo apt-get install yasm
3.安装sdl1.2
sudo apt-get install libsdl1.2-dev
4.安装 sdl2.0
sudo apt-get install libsdl2-dev
5.编译安装ffmpeg
进入到解压之后的 ffmpeg文件夹，依次执行以下命令：
./configure
make
sudo make install
6.测试是否安装成功
ffmpeg -version
ffplay -version
7.ffmpeg简单使用
ffmpeg 抽取视频帧 https://blog.csdn.net/weixin_43804210/article/details/107964643
ffmpeg 压缩视频 https://blog.csdn.net/weixin_43804210/article/details/108109386
#.连续帧图像转视频，也可以使用opencv-python。可参考连接：
https://blog.csdn.net/kxh123456/article/details/121692474
（该方法可能读取的连续帧图像是乱序的，需要自己加个排序代码。想要完整代码的可以私信我）
1.读入第一帧
代码如下（示例）track.py line55：
for path, img, img0 in dataloader:
timer.tic()
blob = torch.from_numpy(img).cuda().unsqueeze(0) # ([1, 3, 608, 1088])
online_targets = tracker.update(blob, img0)
online_targets 已经得到56个结果，后面是结果可视化。下面看tracker.update做了什么。
1.检测
主要得到三个结果，hm(热图，2维)、wh(2维)、id_feature(512维)、reg(偏移量，2维)
在4倍下采样特征图上。随后经过后处理等操作，得到56个检测目标
output = self.model(im_blob)[-1] # (1,1,152,272 ) (1,2,152,272 ) (1,512,152,272 ) (1,2,152,272 )
hm = output['hm'].sigmoid_()
wh = output['wh']
id_feature = output['id']
id_feature = F.normalize(id_feature, dim=1)
reg = output['reg'] if self.opt.reg_offset else None
dets, inds = mot_decode(hm, wh, reg=reg, cat_spec_wh=self.opt.cat_spec_wh, K=self.opt.K) # (1,128,6) (1,128)
dets = self.post_process(dets, meta) # (1,128,6) --> dets[1]: ( 128,5 )
dets = self.merge_outputs([dets])[1]
2.跟踪
依次经过以下2步。
Step 2: First association, with embedding
Step 3: Second association, with IOU
由于此前 self.tracked_stracks 与 self.lost_stracks 为空，以上两步省略，直接创建新目标
for inew in u_detection:
track = detections[inew]
if track.score < self.det_thresh:
continue
track.activate(self.kalman_filter, self.frame_id)
activated_starcks.append(track)
这里的u_detection 与 detections就是刚才的检测结果。activate代码如下：
def activate(self, kalman_filter, frame_id):
"""Start a new tracklet"""
self.kalman_filter = kalman_filter
self.track_id = self.next_id()
self.mean, self.covariance = self.kalman_filter.initiate(self.tlwh_to_xyah(self._tlwh)) # (8) (8,8)
self.tracklet_len = 0
self.state = TrackState.Tracked # 1
if frame_id == 1:
self.is_activated = True
#self.is_activated = True
self.frame_id = frame_id
self.start_frame = frame_id
然后就是可视化。这里第一帧检测结果就开始跟踪了，新创建的tracker: state=True，并且is_activated（*到后面的新检测结果就不激活了，这里注意 *）.
2.第二帧
1.对第二帧检测
这里省略，代码同上。检测得到57个结果：dets、 detections、inds
2.跟踪
unconfirmed = []
tracked_stracks = [] # type: list[STrack]
for track in self.tracked_stracks: # 上一帧的56个结果
if not track.is_activated:
unconfirmed.append(track) # 不执行
else:
tracked_stracks.append(track) # 执行
Step 2: First association, with embedding ：目标间的特征距离
1.dists = matching.embedding_distance(strack_pool, detections)
strack_pool = joint_stracks(tracked_stracks, self.lost_stracks)
# Predict the current location with KF
#for strack in strack_pool:
#strack.predict()
STrack.multi_predict(strack_pool) # 更新卡尔曼滤波参数(更新后的mean conv等于旧的)
dists = matching.embedding_distance(strack_pool, detections)
# 计算56个跟踪结果和57个检测结果，对应特征的余弦距离。计算代码如下：
matching.***embedding_distance***：计算跟踪目标与新检测目标的余弦距离：
def embedding_distance(tracks, detections, metric='cosine'):
cost_matrix = np.zeros((len(tracks), len(detections)), dtype=np.float)
det_features = np.asarray([track.curr_feat for track in detections], dtype=np.floa # (57,512)
track_features = np.asarray([track.smooth_feat for track in tracks], dtype=np.float) # (56,512)
cost_matrix = np.maximum(0.0, cdist(track_features, det_features, metric)) # Nomalized features
return cost_matrix # （56，57）
2.dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)
def fuse_motion(kf, cost_matrix, tracks, detections, only_position=False, lambda_=0.98):
if cost_matrix.size == 0:
return cost_matrix
gating_dim = 2 if only_position else 4 # 4
gating_threshold = kalman_filter.chi2inv95[gating_dim] # 9.48
measurements = np.asarray([det.to_xyah() for det in detections]) # ( 57,4 )
for row, track in enumerate(tracks):
gating_distance = kf.gating_distance(
track.mean, track.covariance, measurements, only_position, metric='maha')
# 门距离，根据预测值(跟踪器)与观测值(检测结果)的位置和距离计算，维度（57）
cost_matrix[row, gating_distance > gating_threshold] = np.inf
cost_matrix[row] = lambda_ * cost_matrix[row] + (1 - lambda_) * gating_distance
# 按照0.98和0.02的权重分配： 余弦距离与gating_distance
return cost_matrix #(56, 57)
3.matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7)
matchs有56个，u_track无，u_detection有一个
def linear_assignment(cost_matrix, thresh):
matches, unmatched_a, unmatched_b = [], [], []
cost, x, y = lap.lapjv(cost_matrix, extend_cost=True, cost_limit=thresh)
# cost:1.63 x:(56) y:(57)
for ix, mx in enumerate(x):
if mx >= 0:
matches.append([ix, mx])
unmatched_a = np.where(x < 0)[0] # 无
unmatched_b = np.where(y < 0)[0] # [56]：第56个值为-1，没有找到匹配目标
matches = np.asarray(matches)
return matches, unmatched_a, unmatched_b # （56，2） （） [56]
lap.lapjv：做匈牙利匹配。输入是一个n*m得分矩阵，返回三个值：
c: 赋值的代价(越小越好)，如果return_cost为False，则不返回。
x: 一个大小为n的数组，用于指定每一行被分配到哪一列。
y: 一个大小为m的数组，用于指定每列被分配到哪一行。
5.
for itracked, idet in matches: # 循环56个已匹配目标
track = strack_pool[itracked]
det = detections[idet]
if track.state == TrackState.Tracked:
track.update(detections[idet], self.frame_id) # 根据匹配结果，更新卡尔曼滤波(包括特征值)
activated_starcks.append(track)
else:
track.re_activate(det, self.frame_id, new_id=False)
refind_stracks.append(track)
Step 3: Second association, with IOU
u_track为空，因此r_tracked_stracks为空，dists为空。
detections = [detections[i] for i in u_detection] # ( 56 ) * OT_O_(0-0)
r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked] # []
dists = matching.iou_distance(r_tracked_stracks, detections) # []
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5) # [] () (0)
Step 3.5：Deal with unconfirmed tracks
没做任何操作，因为 unconfirmed 为空
detections = [detections[i] for i in u_detection] # 【56】
dists = matching.iou_distance(unconfirmed, detections) # unconfirmed为[],dists为[]
matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7) # [] () (0)
Step 4: Init new stracks
为 u_detection 中的一个目标创建跟踪器。
for inew in u_detection:
track = detections[inew]
if track.score < self.det_thresh:
continue
track.activate(self.kalman_filter, self.frame_id)
activated_starcks.append(track)
Step 5: Update state
没执行，因为 self.lost_stracks 为空
3.更新整个跟踪器
self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked] # 56个（之前的）
self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks) # 57个（加上新的检测）
self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks) # 无增加：refind_stracks为空
self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks) # 无
self.lost_stracks.extend(lost_stracks) # 无
self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks) # 无
self.removed_stracks.extend(removed_stracks) # 无
self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
output_stracks = [track for track in self.tracked_stracks if track.is_activated] # （56）个
3.第三帧
检测结果为55个目标
detections = [STrack(STrack.tlbr_to_tlwh(tlbrs[:4]), tlbrs[4], f, 30) for
(tlbrs, f) in zip(dets[:, :5], id_feature)] # 55
unconfirmed = []
tracked_stracks = [] # type: list[STrack]
for track in self.tracked_stracks: # 上一帧57个目标
if not track.is_activated:
unconfirmed.append(track) # 一个目标
else:
tracked_stracks.append(track) # 上一帧56个目标
Step 2: First association, with embedding
strack_pool = joint_stracks(tracked_stracks, self.lost_stracks) # 56
STrack.multi_predict(strack_pool)
dists = matching.embedding_distance(strack_pool, detections) # ( 56,55)
#dists = matching.gate_cost_matrix(self.kalman_filter, dists, strack_pool, detections)
dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections)
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7) # (55，2) [52] []
然后对 matches, u_track, u_detection 依次处理
更新已匹配的55个跟踪器：
for itracked, idet in matches:
track = strack_pool[itracked]
det = detections[idet]
if track.state == TrackState.Tracked:
track.update(detections[idet], self.frame_id)
activated_starcks.append(track) # 55个
else:
track.re_activate(det, self.frame_id, new_id=False)
refind_stracks.append(track) # 无
Step 3: Second association, with IOU
计算 多出(未配对)的检测结果，与多出(未配对)的跟踪器，之间的iou距离：
detections = [detections[i] for i in u_detection] # 空：多出(未配对)的检测结果
r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked] # u_track：[52]
dists = matching.iou_distance(r_tracked_stracks, detections) # []
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5) # （） (0) （）
处理未配对的跟踪器（跟踪结果未与检测结果匹配）
for it in u_track:
track = r_tracked_stracks[it] # 上一帧未匹配的那一个跟踪器
if not track.state == TrackState.Lost: # 执行
track.mark_lost() # track.state = 2
lost_stracks.append(track) # lost_stracks由 [] 增加一个
处理上一帧未激活的跟踪器（也就是第二帧中(57)，未与第一帧(56)配对的那一个检测结果）
detections = [detections[i] for i in u_detection] # []
dists = matching.iou_distance(unconfirmed, detections) # []
matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7) # () (0) ()
for it in u_unconfirmed:
track = unconfirmed[it]
track.mark_removed() # track.state = 1 --> =3 状态转为3
removed_stracks.append(track) # 由[] 添加1
Step 4: Init new stracks ：由于u_detection为空，不执行
Step 5: Update state 由于self.lost_stracks为空，不执行
更新整个跟踪器
self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked] # 55个
self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks) # 55个
self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks) # 55个 无增加：refind_stracks为空
self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks) # 无
self.lost_stracks.extend(lost_stracks) # 增一个：未匹配的跟踪[OT_53_(1-2)]
self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks) # 无增加，还是1个
self.removed_stracks.extend(removed_stracks) # 一个:第2帧中多出来的检测
self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
# （55）（1）
output_stracks = [track for track in self.tracked_stracks if track.is_activated] # （55）个
4.第四帧
检测结果为55个目标
每次跟踪开始前，先初始化：
self.frame_id += 1
activated_starcks = []
refind_stracks = []
lost_stracks = []
removed_stracks = []
unconfirmed 没有，tracked_stracks有55个：
unconfirmed = []
tracked_stracks = []
for track in self.tracked_stracks:
if not track.is_activated:
unconfirmed.append(track)
else:
tracked_stracks.append(track)
Step 2: First association, with embeddig
strack_pool有56个（添加了上一帧中的self.lost_stracks）
strack_pool = joint_stracks(tracked_stracks, self.lost_stracks) # 56
STrack.multi_predict(strack_pool)
dists = matching.embedding_distance(strack_pool, detections) # ( 56,55)
dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections) # ( 56,55)
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7) # (55，2) [34] []
处理配对的55个跟踪器：
activated_starcks：54个 ，refind_stracks：1个（[53]）
for itracked, idet in matches:
track = strack_pool[itracked]
det = detections[idet]
if track.state == TrackState.Tracked:
track.update(detections[idet], self.frame_id)
activated_starcks.append(track)
else:
track.re_activate(det, self.frame_id, new_id=False)
refind_stracks.append(track) # 上一帧的self.lost_stracks
Step 3: Second association, with IOU
detections 为空，u_track 有1个，r_tracked_stracks 1个：
detections = [detections[i] for i in u_detection] # ( 56 ) * OT_O_(0-0)
r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked] # []
dists = matching.iou_distance(r_tracked_stracks, detections) # []
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5) # [] (0) ()
对u_track（为跟检测结果匹配）的跟踪器，标为lost：
for it in u_track:
track = r_tracked_stracks[it]
if not track.state == TrackState.Lost:
track.mark_lost()
lost_stracks.append(track)
Deal with unconfirmed tracks, usually tracks with only one beginning frame
Step 4: Init new stracks
因为没有u_detection，这步跳过，直接第5步骤：
Step 5: Update state
self.lost_stracks是上一帧的未匹配跟踪器，不是刚才的lost_stracks：
for track in self.lost_stracks:
if self.frame_id - track.end_frame > self.max_time_lost: # 判断是否丢失超过15秒
track.mark_removed()
removed_stracks.append(track) # 未执行
更新整个跟踪器
self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked] # 54
self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks) # 54
self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks) # 55
self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks) # []
self.lost_stracks.extend(lost_stracks) # 1个：【35】
self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks) # 1个：【35】
self.removed_stracks.extend(removed_stracks) # 1个 [57]
self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks) # (55) [35]
# get scores of lost tracks
output_stracks = [track for track in self.tracked_stracks if track.is_activated] # [55]
因为上一帧的self.lost_stracks，在这帧已经匹配了，所以变成了refind_stracks
5.第5帧
检测结果为56个目标
上一帧跟踪器为55个，unconfirmed为0
unconfirmed = []
tracked_stracks = []
for track in self.tracked_stracks:
if not track.is_activated:
unconfirmed.append(track)
else:
tracked_stracks.append(track)
Step 2: First association, with embedding
strack_pool = joint_stracks(tracked_stracks, self.lost_stracks) # 56
STrack.multi_predict(strack_pool)
dists = matching.embedding_distance(strack_pool, detections) # ( 56,56)
dists = matching.fuse_motion(self.kalman_filter, dists, strack_pool, detections) # ( 56,56)
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.7) # (55，2) [55] [48]
更新已匹配的55个match：
for itracked, idet in matches:
track = strack_pool[itracked]
det = detections[idet]
if track.state == TrackState.Tracked:
track.update(detections[idet], self.frame_id)
activated_starcks.append(track) # 55 个
else:
track.re_activate(det, self.frame_id, new_id=False)
refind_stracks.append(track) # 【】
Step 3: Second association, with IOU
detections = [detections[i] for i in u_detection] # OT_O_(0-0)
r_tracked_stracks = [strack_pool[i] for i in u_track if strack_pool[i].state == TrackState.Tracked] # []
dists = matching.iou_distance(r_tracked_stracks, detections) # []
matches, u_track, u_detection = matching.linear_assignment(dists, thresh=0.5) # [] () (0)
'Deal with unconfirmed tracks, usually tracks with only one beginning frame
detections = [detections[i] for i in u_detection] # 1个
dists = matching.iou_distance(unconfirmed, detections) # []
matches, u_unconfirmed, u_detection = matching.linear_assignment(dists, thresh=0.7) # [] () (0)
for itracked, idet in matches: # 不执行
unconfirmed[itracked].update(detections[idet], self.frame_id)
activated_starcks.append(unconfirmed[itracked])
for it in u_unconfirmed:
track = unconfirmed[it]
track.mark_removed() # track.state = 1 --> track.state = 3
removed_stracks.append(track)
Step 4: Init new stracks
activated_starcks 增加1，其state = 1 ：
for inew in u_detection:
track = detections[inew]
if track.score < self.det_thresh:
continue
track.activate(self.kalman_filter, self.frame_id)
activated_starcks.append(track)
更新整个跟踪器
self.tracked_stracks = [t for t in self.tracked_stracks if t.state == TrackState.Tracked] # 55
self.tracked_stracks = joint_stracks(self.tracked_stracks, activated_starcks) # 56
self.tracked_stracks = joint_stracks(self.tracked_stracks, refind_stracks) # 55
self.lost_stracks = sub_stracks(self.lost_stracks, self.tracked_stracks)
self.lost_stracks.extend(lost_stracks)
self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)
self.removed_stracks.extend(removed_stracks)
self.tracked_stracks, self.lost_stracks = remove_duplicate_stracks(self.tracked_stracks, self.lost_stracks)
# get scores of lost tracks
output_stracks = [track for track in self.tracked_stracks if track.is_activated] # 55个（其中有个跟踪器，状态是未激活的）