RuntimeError: CUDA error: invalid configuration argument CUDA kernel errors might be asynchronously

RuntimeError: CUDA error: invalid configuration argument CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1解决指南

一、废话

我在训练Rank-DETR时只开了一张Nvidia-A800显卡，batchsize=2；然后训练到低n轮时报了如下错误：

RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

网上查了不少方法都未曾奏效，后来我突然发现爆显存也会触发如上错误，故我讲batchsize修改为1后解决了问题。

二、解决方案

尝试减小batchsize