pytorch sets random seed to exclude randomness

---

This article does not agree to reprint, and it is forbidden to reprint in any form! !

Preface

Setting up a random seed is very important for repetitive experiments or comparative experiments. The official website of pytorch also provides documentation .

Set random seed

In order to solve the randomness, it is necessary to restrict all the places that generate randomness. Here I summarized it myself:

1. Exclude PyTorch's randomness
2. Exclude the randomness of third-party libraries
3. Exclude the randomness of cudnn acceleration

This is the method given by mmdetection:

def set_random_seed(seed, deterministic=False):
    """Set random seed.

    Args:
        seed (int): Seed to be used.
        deterministic (bool): Whether to set the deterministic option for
            CUDNN backend, i.e., set `torch.backends.cudnn.deterministic`
            to True and `torch.backends.cudnn.benchmark` to False.
            Default: False.
    """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    if deterministic:
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False

The cuDNN library used by CUDA convolution operations may be a source of uncertainty for multiple executions across applications. When calling cuDNN convolution with a new set of size parameters, an optional feature can run multiple convolution algorithms and benchmark them to find the fastest one. Then, in the rest of the process, for the corresponding size parameter set, the fastest algorithm will be used consistently. Due to the noise of the benchmark test and different hardware, the benchmark test may choose different algorithms in subsequent runs, even on the same machine . You can set torch.backends.cudnn.benchmark = False. Disabling the benchmark function will cause cuDNN to select algorithms deterministically, possibly at the cost of performance degradation.

torch.use_deterministic_algorithms() allows you to configure PyTorch to use deterministic algorithms when available, instead of non-deterministic algorithms. If the operation is known to be non-deterministic (and there is no deterministic alternative), an error will be thrown.

Although the CUDA convolution benchmark is disabled to ensure that CUDA selects the same algorithm application every time it runs, the algorithm itself may be uncertain, unless torch.use_deterministic_algorithms(true) or torch.backends.cudnn.deterministic = True . The latter is only set to control this behavior, and torch.use_deterministic_algorithms() will make the behavior of other PyTorch operations deterministic.

def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True / False

DataLoader

If you find that the random seed is fixed when training the network, but the setting of num_workers in torch.utils.data.DataLoader greater than 0 will also cause the two training results to be different. This should be caused by the inconsistent reading order of the multi-process. After I set it to 0, the result is the same, but the speed drops a lot. Solution:

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

DataLoader(
    train_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    worker_init_fn=seed_worker
)

This article does not agree to reprint, and it is forbidden to reprint in any form! !