CUDA apps time out & fail after several seconds - how to work around this?

Question

I\'ve noticed that CUDA applications tend to have a rough maximum run-time of 5-15 seconds before they will fail and exit out. I realize it\'s ideal to not have CUDA applic

Answer 1

The solution I use is:

1. Pass all information to device.
2. Run iterative versions of algorithms, where each iteration invokes the kernel on the memory already stored within the device.
3. Finally transfer memory to host only after all iterations have ended.

This enables control over iterations from CPU (including option to abort), without the costly device<-->host memory transfers between iterations.

Answer 2

It is possible to disable this behavior in Linux. Although the "watchdog" has an obvious purpose, it may cause some very unexpected results when doing extensive computations using shaders / CUDA.

The option can be toggled in your X-configuration (likely /etc/X11/xorg.conf)

Adding: Option "Interactive" "0" to the device section of your GPU does the job.

see CUDA Visual Profiler 'Interactive' X config option?

For details on the config

and

see ftp://download.nvidia.com/XFree86/Linux-x86/270.41.06/README/xconfigoptions.html#Interactive

For a description of the parameter.