How to make sure Chainer Training is using same device for operations?

Question

I am training a chainer model in GeForce 2080 Ti. 5 out of 10 trainings fail and throw cupy-cuda error.

chainer.cuda.set_max_workspace_size(1024 * 1024 * 1024)
ch