Caltech-256 数据集处理(二) 训练集和测试集的制作

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/tfcy694/article/details/84259556

Caltech-256数据集在PyTorch中的处理：
Caltech-256 数据集处理(一) label提取
Caltech-256 数据集处理(二) 训练集和测试集的制作
Caltech-256 数据集处理(三) 训练集和验证集载入Dateloader

第一篇文章提取了label，后面做实验发现还是不够用，这里用10：1的比例划分训练集和验证集，可以基本保证不发生类别不平衡的情况。原图片结构依然同第一篇：
./256_ObjectCategories/001.ak47/001_0001.jpg
… …
./256_ObjectCategories/257.clutter/257_0827.jpg

import os

datasetpath="/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories"

dirs=os.listdir(datasetpath)
dirs.sort()
with open(r'label.txt','w',encoding='utf-8') as f:
    for i in dirs:
        f.write(i)
        f.write('\n')

#print(dirs)
it=0
Matrix = [[] for x in range(257)]                # all filenames under DATA_PATH
for d in dirs:
    for _, _, filename in os.walk(os.path.join(datasetpath,d)):
        for i in filename:
            Matrix[it].append(os.path.join(os.path.join(datasetpath,d),i))  # filename is a list of pic files under the fold
    it = it + 1

#print(Matrix)
with open(r'dataset-val.txt','w',encoding='utf-8') as f:
    for i in range(len(Matrix)):
        for j in range(10):
            f.write(os.path.join(datasetpath,Matrix[i][j]))
            f.write(' ')
            f.write(str(i))
            f.write('\n')
with open(r'dataset-trn.txt','w',encoding='utf-8') as f:
    for i in range(len(Matrix)):
        for j in range(10,len(Matrix[i])):
            f.write(os.path.join(datasetpath,Matrix[i][j]))
            f.write(' ')
            f.write(str(i))
            f.write('\n')

生成的带标签的训练集和验证集部分如下：
dataset-trn.txt

/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0010.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0040.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0091.jpg 0
... ...
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0056.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0030.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0011.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0052.jpg 1
... ...
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0774.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0233.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0013.jpg 256

dataset-val.txt

/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0070.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0060.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0076.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0062.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0035.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0092.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0015.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0098.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0024.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/001.ak47/001_0003.jpg 0
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0078.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0093.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0037.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0065.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0055.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0069.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0002.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0012.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0096.jpg 1
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/002.american-flag/002_0032.jpg 1
... ...
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0061.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0365.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0543.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0530.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0556.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0310.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0531.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0305.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0135.jpg 256
/media/this/02ff0572-4aa8-47c6-975d-16c3b8062013/256_ObjectCategories/257.clutter/257_0116.jpg 256