Getting “CancelledError: [_Derived_]RecvAsync is cancelled” after hours of training

Question

I am getting the following error during training on multiple GPUs.

The weird thing here is that the training runs for quite some time without any problems but after a