NumbaPro on Cuda device over ssh connection

Question

I'm using Python/NumbaPro to use my CUDA complient GPU on a windows box. I use Cygwin as shell and from within a cygwin console it has no problems finding my CUDA device. I test with the simple command

    numbapro.check_cuda()

But when I'm connection to the box over OpenSSH (as part of my Cygwin setup), I get the following error:

numba.cuda.cudadrv.error.CudaSupportError: Error at driver init:
Call to cuInit results in CUDA_ERROR_NO_DEVICE:

How to fix this?

Answer 1

The primary cause of this is Windows service session 0 isolation. When you run any application via a service which runs in session 0 (so sshd, or windows remote desktop, for example), the machines native display driver is unavailable. For CUDA applications, this means that you are get a no device available error at runtime because the sshd you use to login is running as a service and there is no available CUDA driver.

The are a few workarounds:

1. Run the sshd as a process rather than a service.
2. If you have a compatible GPU, use the TCC driver rather than the GPU display driver.

On the secondary problem, the Python runtime error you are seeing comes from the multiprocessing module. From this question it appears that the root cause is probably the NUMBER_OF_PROCESSORS environment variable not being set. You can use one of the workarounds in that thread to get around that problem