Article directory
1. numpy library operations
numpy.save()
: Arrays are saved in uncompressed raw binary format in files with a .npy extension.numpy.load()
: Read the data of the .npy file and directly convert it to a numpy array
☀☀☀<<example>>☀☀☀
>>> import numpy as np
>>> a = np.arange(24).reshape(2,3,4)
>>> a
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
>>> np.save("D:/aa/npp.npy", a)
>>> b = np.load("D:/aa/npp.npy")
>>> b
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
If the error Object arrays cannot be loaded when allow_pickle=False is reported , it needs to be changed as follows:
np.load(path, allow_pickle=True)
2. pandas library operation
to_pickle
: This method is to directly store the dataframe variable data as a local file, and there is no requirement for the file extensionread_pickle
: This method is to read the variables stored locally and convert them into dataframe files
☀☀☀<<example>>☀☀☀
>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame(np.arange(12).reshape(3, 4))
>>> df
0 1 2 3
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
>>> df.columns = ['I', 'II', 'III', 'IV']
>>> df
I II III IV
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
>>> df.to_pickle(r"D:/aa/df")
>>> dff = pd.read_pickle(r"D:/aa/df")
>>> dff
I II III IV
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
3. sklearn library operation
After testing, the saved file size numpy.save()
is consistent with the method.
☀☀☀<<example>>☀☀☀
Note: The new version of sklearn will report an error, just import joblib directly
Reference: (note the comment below) cannot import name joblib from sklearn.externals
from sklearn.externals import joblib
# 保存x
joblib.dump(x, 'x.pkl')
# 加载x
x = joblib.load('x.pkl')
The new implementation method
import joblib
# 保存x
joblib.dump(x, 'x.pkl')
# 加载x
x = joblib.load('x.pkl')
4. Pickle library operation
☀☀☀<<example>>☀☀☀
import pickle
# 存储变量的文件的名字
filename = 'shoplist.data'
# 初始化变量
shoplist = ['apple', 'mango', 'carrot']
# 以二进制写模式打开目标文件
f = open(filename, 'wb')
# 将变量存储到目标文件中区
pickle.dump(shoplist, f)
# 关闭文件
f.close()
# 删除变量
del shoplist
# 以二进制读模式打开目标文件
f = open(filename, 'rb')
# 将文件中的变量加载到当前工作区
storedlist = pickle.load(f)
print(storedlist)
5. Pytorch library operation
Model saving and loading method 1:
keep:
torch.save(model.state_dict(), mymodel.pth) # 只保存模型权重参数,不保存模型结构
transfer:
model = My_model(*args, **kwargs) #这里需要重构模型结构,My_model
model.load_state_dict(torch.load(mymodel.pth)) #这里根据模型结构,调用存储的模型参数
model.eval()
Model saving and loading method two:
keep:
torch.save(model, mymodel.pth) # 保存整个 model 的状态
transfer:
model=torch.load(mymodel.pth) # 这里已经不需要重构模型结构了,直接 load 就可以
model.eval()