版权声明：学无止境，好好学习 https://blog.csdn.net/m0_38116269/article/details/88327203
前言：
前段时间自己上手之间训练给定的数据集，发现训练精度提高到70%左右就不变了，苦苦调参终究不是王道，准备给它来个曲线救国。到底是我模型的问题？还是数据集的问题，目前我不确定。所以准备另起炉灶，重新制作数据集，从无到有，一步一步实现物体识别。准备开始。
一、制作数据集
毕竟这是实验阶段，尽量准备容易判断的数据集进行实验。所以，咱们准备做一个动物分类的数据集。里面有六类动物数据。训练集每类大约450张图片，测试集每类50张图片。制作过程就是用Python爬去百度图片的数据。
实在是太菜了，还得借鉴别人的代码，正则表达式写法还不会，好歹最终是做出来了。
代码如下：
import re
import requests
#这个是网址，找个百度图片地址复制上即可。
url = 'https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1552035756177_R&pv=&ic=0&nc=1&z=&hd=&latest=©right=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E9%A9%AC'
t=0
i=1
#准备爬取500张图片。说道五百，孙悟空压在五行山下也是五百年，今年下半年......文体两开花......
while t < 500:
urll=url+str(t)+'&gsm=8c'
html = requests.get(urll).text
pic_url = re.findall('"thumbURL":"(.*?)",', html, re.S)
for each in pic_url:
print('正在下载第' + str(i + 1) + '张图片，图片地址:' + str(each))
try:
pic= requests.get(each, timeout=10)
except requests.exceptions.ConnectionError:
print('【错误!】当前图片无法下载')
continue
string = 'W:\\picture\\train\\c3horse\\' + "horse_" + str(i) + '.jpg' #自己命名即可，目录要注意设置
fp = open(string, 'wb')
fp.write(pic.content)
fp.close()
i += 1
t += 1
我找了六类，最终如下：这是训练集和测试集
曾经出现的问题：
1.
File "D:\Anaconda\lib\urllib\request.py", line 258, in urlretrieve
tfp = open(filename, 'wb')
OSError: [Errno 22] Invalid argument: 'W:\\picture\train\\cat0.jpg'
原因：自己的路径\train这个与\t制表符发生冲突，因此需要改动改成反斜杠就不会冲突了。
2.
报错：
python3.6报错：AttributeError: ‘str’ object has no attribute ‘decode’
解决：把decode改为encode即可。
参考：
caffe学习笔记（四）–制作自己的数据集train.txt和val.txt，生成LMDB文件
有点用
这个博客好
二、给数据集加标签
这也是重要的一步，因为训练是读取数据就是通过读取txt文件进行的。因此需要建立train.txt和test.txt
注意：创建的时候，txt文件中应该给出的是相对路径，或者说直接读取数据可能可以用相对路径？因为不这样会出现如下错误：
路径错误，所以找不到。
wrong:
seu@seu-HP-Z640:/media/seu/Linux-doc/wwh/caffe$ sh ./data/animal_picture/create_lmdb.sh
Creating train lmdb...
I0308 21:03:00.933708 10679 convert_imageset.cpp:86] Shuffling data
I0308 21:03:01.937492 10679 convert_imageset.cpp:89] A total of 2699 images.
F0308 21:03:01.937552 10679 db_lmdb.cpp:13] Check failed: mkdir(source.c_str(), 0744) == 0 (-1 vs. 0) mkdir data/animal_picture/animal_train_lmdb failed
*** Check failure stack trace: ***
@ 0x7f071f7415cd google::LogMessage::Fail()
@ 0x7f071f743433 google::LogMessage::SendToLog()
@ 0x7f071f74115b google::LogMessage::Flush()
@ 0x7f071f743e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f071fcbe6e8 caffe::db::LMDB::Open()
@ 0x403aaf main
@ 0x7f071e6b1830 __libc_start_main
@ 0x404b79 _start
@ (nil) (unknown)
Aborted (core dumped)
这是因为没建立好lmdb文件之前这个目录就存在了，我需要删除就可以了。
another wrong:
seu@seu-HP-Z640:/media/seu/Linux-doc/wwh/caffe$ sh ./data/animal_picture/create_lmdb.sh
Creating train lmdb...
I0308 21:07:35.976945 11569 convert_imageset.cpp:86] Shuffling data
I0308 21:07:36.951099 11569 convert_imageset.cpp:89] A total of 2699 images.
I0308 21:07:36.951357 11569 db_lmdb.cpp:35] Opened lmdb data/animal_picture/animal_train_lmdb
E0308 21:07:36.978814 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_349.jpg
E0308 21:07:37.081532 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_52.jpg
E0308 21:07:37.159948 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_59.jpg
E0308 21:07:37.207358 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c3horse/horse_274.jpg
E0308 21:07:37.308380 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_128.jpg
E0308 21:07:37.440773 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c4rabbit/rabbit_321.jpg
E0308 21:07:37.563056 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c4rabbit/rabbit_413.jpg
E0308 21:07:37.642185 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c4rabbit/rabbit_123.jpg
E0308 21:07:37.732091 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c0bird/bird_219.jpg
E0308 21:07:37.816795 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_265.jpg
E0308 21:07:38.164516 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_202.jpg
E0308 21:07:38.467152 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_301.jpg
E0308 21:07:38.475965 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_299.jpg
I0308 21:07:39.884591 11569 convert_imageset.cpp:147] Processed 1000 files.
E0308 21:07:40.088840 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c3horse/horse_102.jpg
E0308 21:07:40.216799 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_336.jpg
E0308 21:07:40.557474 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_263.jpg
E0308 21:07:41.024181 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_269.jpg
E0308 21:07:41.129235 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c5tiger/tiger_137.jpg
E0308 21:07:41.162933 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_283.jpg
E0308 21:07:41.448032 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_206.jpg
E0308 21:07:41.650552 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_328.jpg
I0308 21:07:42.817777 11569 convert_imageset.cpp:147] Processed 2000 files.
E0308 21:07:42.902566 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c5tiger/tiger_186.jpg
E0308 21:07:43.127310 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_506.jpg
E0308 21:07:43.346856 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c3horse/horse_347.jpg
E0308 21:07:43.419414 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c4rabbit/rabbit_139.jpg
E0308 21:07:43.506410 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_299.jpg
E0308 21:07:43.663766 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c1cat/cat_289.jpg
E0308 21:07:43.935115 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_359.jpg
E0308 21:07:43.979183 11569 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/train/c2dog/dog_262.jpg
I0308 21:07:44.850670 11569 convert_imageset.cpp:153] Processed 2670 files.
Creating val lmdb...
I0308 21:07:45.028161 11606 convert_imageset.cpp:86] Shuffling data
I0308 21:07:45.975391 11606 convert_imageset.cpp:89] A total of 300 images.
I0308 21:07:45.975666 11606 db_lmdb.cpp:35] Opened lmdb data/animal_picture/animal_test_lmdb
E0308 21:07:46.099419 11606 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/test/dog_16.jpg
E0308 21:07:46.221585 11606 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/test/horse_9.jpg
E0308 21:07:46.382019 11606 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/test/cat_39.jpg
E0308 21:07:46.494719 11606 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/test/horse_31.jpg
E0308 21:07:46.496433 11606 io.cpp:80] Could not open or find file /media/seu/Linux-doc/wwh/caffe/data/animal_picture/test/dog_9.jpg
I0308 21:07:46.910825 11606 convert_imageset.cpp:153] Processed 295 files.
Done.
这个图片坏掉了，我应该删除它。
succeed!!!
seu@seu-HP-Z640:/media/seu/Linux-doc/wwh/caffe$ sh ./data/animal_picture/create_lmdb.sh
Creating train lmdb...
I0308 21:27:40.091610 15824 convert_imageset.cpp:86] Shuffling data
I0308 21:27:41.056541 15824 convert_imageset.cpp:89] A total of 2667 images.
I0308 21:27:41.056802 15824 db_lmdb.cpp:35] Opened lmdb data/animal_picture/animal_train_lmdb
I0308 21:27:44.071780 15824 convert_imageset.cpp:147] Processed 1000 files.
I0308 21:27:46.963354 15824 convert_imageset.cpp:147] Processed 2000 files.
I0308 21:27:48.912989 15824 convert_imageset.cpp:153] Processed 2667 files.
Creating val lmdb...
I0308 21:27:49.111166 15858 convert_imageset.cpp:86] Shuffling data
I0308 21:27:50.061434 15858 convert_imageset.cpp:89] A total of 295 images.
I0308 21:27:50.061682 15858 db_lmdb.cpp:35] Opened lmdb data/animal_picture/animal_test_lmdb
I0308 21:27:50.988663 15858 convert_imageset.cpp:153] Processed 295 files.
Done.
最终文件如下：
三、制作均值文件
这一步是数据预处理当中的选项。运行代码和结果如下。
seu@seu-HP-Z640:/media/seu/Linux-doc/wwh/caffe$ ./build/tools/compute_image_mean data/animal_picture/animal_train_lmdb data/animal_picture/animal_mean.binaryproto
I0308 22:00:24.008167 22011 db_lmdb.cpp:35] Opened lmdb data/animal_picture/animal_train_lmdb
I0308 22:00:24.009119 22011 compute_image_mean.cpp:70] Starting iteration
I0308 22:00:24.351930 22011 compute_image_mean.cpp:101] Processed 2667 files.
I0308 22:00:24.352077 22011 compute_image_mean.cpp:108] Write to data/animal_picture/animal_mean.binaryproto
I0308 22:00:24.353020 22011 compute_image_mean.cpp:114] Number of channels: 3
I0308 22:00:24.353073 22011 compute_image_mean.cpp:119] mean_value channel [0]: 125.581
I0308 22:00:24.353142 22011 compute_image_mean.cpp:119] mean_value channel [1]: 143.585
I0308 22:00:24.353194 22011 compute_image_mean.cpp:119] mean_value channel [2]: 148.823
四、利用卷积神经网络模型进行训练
先上一个resnet-18试试效果如何。
我的文件包括solver.prototxt, train_test.prototxt, train_resnet.sh。
I0308 22:30:54.353442 26237 data_layer.cpp:73] Restarting data prefetching from start.
I0308 22:30:54.369786 26212 solver.cpp:414] Test net output #0: acc/top-1 = 0.864407
I0308 22:30:54.369805 26212 solver.cpp:414] Test net output #1: acc/top-2 = 0.952542
I0308 22:30:54.369812 26212 solver.cpp:414] Test net output #2: loss = 0.458069 (* 1 = 0.458069 loss)
I0308 22:30:54.420521 26212 solver.cpp:239] Iteration 9600 (15.4126 iter/s, 6.48821s/100 iters), loss = 0.0217188
I0308 22:30:54.420554 26212 solver.cpp:258] Train net output #0: loss = 0.021719 (* 1 = 0.021719 loss)
I0308 22:30:54.420562 26212 sgd_solver.cpp:112] Iteration 9600, lr = 1e-05
I0308 22:30:57.659656 26233 data_layer.cpp:73] Restarting data prefetching from start.
I0308 22:30:59.575147 26212 solver.cpp:239] Iteration 9700 (19.4002 iter/s, 5.1546s/100 iters), loss = 0.0786674
I0308 22:30:59.575183 26212 solver.cpp:258] Train net output #0: loss = 0.0786675 (* 1 = 0.0786675 loss)
I0308 22:30:59.575189 26212 sgd_solver.cpp:112] Iteration 9700, lr = 1e-05
I0308 22:31:04.739707 26212 solver.cpp:347] Iteration 9800, Testing net (#0)
I0308 22:31:06.014310 26237 data_layer.cpp:73] Restarting data prefetching from start.
I0308 22:31:06.030256 26212 solver.cpp:414] Test net output #0: acc/top-1 = 0.867797
I0308 22:31:06.030282 26212 solver.cpp:414] Test net output #1: acc/top-2 = 0.949153
I0308 22:31:06.030293 26212 solver.cpp:414] Test net output #2: loss = 0.433125 (* 1 = 0.433125 loss)
I0308 22:31:06.081120 26212 solver.cpp:239] Iteration 9800 (15.3706 iter/s, 6.50594s/100 iters), loss = 0.0102489
I0308 22:31:06.081178 26212 solver.cpp:258] Train net output #0: loss = 0.010249 (* 1 = 0.010249 loss)
I0308 22:31:06.081193 26212 sgd_solver.cpp:112] Iteration 9800, lr = 1e-05
I0308 22:31:11.254150 26212 solver.cpp:239] Iteration 9900 (19.3312 iter/s, 5.17298s/100 iters), loss = 0.0201467
I0308 22:31:11.254185 26212 solver.cpp:258] Train net output #0: loss = 0.0201468 (* 1 = 0.0201468 loss)
I0308 22:31:11.254192 26212 sgd_solver.cpp:112] Iteration 9900, lr = 1e-05
I0308 22:31:16.278568 26233 data_layer.cpp:73] Restarting data prefetching from start.
I0308 22:31:16.378336 26212 solver.cpp:464] Snapshotting to binary proto file models/animal_classifier/the_resnet_18_iter_10000.caffemodel
I0308 22:31:16.494271 26212 sgd_solver.cpp:284] Snapshotting solver state to binary proto file models/animal_classifier/the_resnet_18_iter_10000.solverstate
I0308 22:31:16.574131 26212 solver.cpp:327] Iteration 10000, loss = 0.0686285
I0308 22:31:16.574159 26212 solver.cpp:347] Iteration 10000, Testing net (#0)
I0308 22:31:17.852732 26237 data_layer.cpp:73] Restarting data prefetching from start.
I0308 22:31:17.869612 26212 solver.cpp:414] Test net output #0: acc/top-1 = 0.874576
I0308 22:31:17.869632 26212 solver.cpp:414] Test net output #1: acc/top-2 = 0.952542
I0308 22:31:17.869640 26212 solver.cpp:414] Test net output #2: loss = 0.436679 (* 1 = 0.436679 loss)
I0308 22:31:17.869644 26212 solver.cpp:332] Optimization Done.
I0308 22:31:17.869647 26212 caffe.cpp:250] Optimization Done.
感觉一般般吧，只达到了87%左右的精度啊。
文件夹中配置文件如下：
其中那个log文件和.caffemodel, .solverstate是最后生成的，用作后面预处理和模型续训用。
五、训练结果可视化（绘制出损失和精度曲线）
之前参考过这个博客，学习绘制训练过程的Loss曲线。但是感觉这个博客的方法好像不太好，原因如下，他利用的Python和ipython notebook弄的，他使用的是mnist数据集，这个数据集非常小，训练几秒就能训练完，训练完绘制图像是及时的，但是我这边训练时间按小时来算，如果在ipython notebook上训练会崩溃，所以不适用。
因此，我准备使用log日志文件，来绘制曲线。
准备参考：https://www.cnblogs.com/carle-09/p/9089718.html
目测不错。
https://blog.csdn.net/u014593748/article/details/76152622
这个是caffe自带的工具：
https://blog.csdn.net/u013989576/article/details/71302244
挺简单，很好用，我的效果如下：
好像不对啊，我的test loss不可能一直上升啊。。。而且我在log里面看也是下降的啊，不知道这串数据怎么出来的。
未完待续…