About pytroch's random number seed

Reasons for training to produce different results

training process

During the training process, the same training set and test set division method, the same weight initialization, and the same hyperparameters, but each training result is different, there may be the following reasons:

1. The presence of dropouts
2. Random seed in PyTorch, Python, Numpy is not fixed
3. The training dataset is randomly shuffled
4. Upsampling and backwards interpolation functions/classes are undefined (PyTorch issue)
   (Notice: Even if all these things are fixed, the results of the model still cannot be completely reproduced because of the difference in hardware equipment. )

the code

def set_seed(seed=1029):
	# 固定python和numpy的随机种子
    random.seed(seed)   # python random seed
    os.environ['PYTHONHASHSEED'] = str(seed)  # hash random seed
    np.random.seed(seed)    # Numpy random seed
	
	# 固定pytorch的随机种子
    torch.manual_seed(seed) # pytorch CPU random seed
    torch.cuda.manual_seed(seed)    # pytorch GPUrandom seed
    torch.cuda.manual_seed_all(seed)  # pytorch if you are using multi-GPU.
    
	# 
    torch.backends.cudnn.benchmark = True
    torch.backends.cudnn.deterministic = True   # Cudnn

cudnn.benchmark

Among them, torch.backends.cudnn.benchmarkif it is True, it will make the program start at the beginningtake a little extra time, search for the most suitable convolution implementation algorithm for each convolutional layer of the entire network, and thenAccelerate the network. It allows the built-in cuDNN auto-tuner to automatically find the most suitable efficient algorithm for the current configuration to achieve the problem of optimizing operating efficiency.

But there are prerequisites for use,The applicable scenario is that the network structure is fixed (not dynamically changing), the input shape of the network (including batch size, image size, and input channel) is unchanged, which is actually more applicable under normal circumstances. Conversely, if the settings of the convolutional layer keep changing and the input data of the network changes every iteration, it will cause cDNN to search for the optimal configuration every time, which will reduce the operating efficiency instead.

cudnn.deterministic

Benchmark mode will increase the calculation speed, but due to the randomness in the calculation, the network feedforward results are slightly different each time. Set to True if you want to avoid this kind of result fluctuation