Tensorflow allocating GPU memory when using tf.device('/cpu:0')

Question

System Info: 1.1.0, GPU, Windows, Python 3.5, code runs in ipython consoles.

I am trying to run two different Tensorflow sessions, one on the GPU (th

Answer 1

Would you mind trying one of these config options ?

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
# or config.gpu_options.per_process_gpu_memory_fraction = 0.0
with tf.Session(config=config) as sess:
    ...

As per the documentation, it should help you manage your GPU memory for this particular session and so your second session should be able to run on GPU.

EDIT: according this answer you should also try this:

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="-1"

Answer 2

It turns out that setting CUDA_VISIBLE_DEVICES to the empty string does not mask the CUDA devices visible to the script.

From the documentation of CUDA_VISIBLE_DEVICES (emphasis added by me):

Only the devices whose index is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. If one of the indices is invalid, only the devices whose index precedes the invalid index are visible to CUDA applications. For example, setting CUDA_VISIBLE_DEVICES to 2,1 causes device 0 to be invisible and device 2 to be enumerated before device 1. Setting CUDA_VISIBLE_DEVICES to 0,2,-1,1 causes devices 0 and 2 to be visible and device 1 to be invisible.

It seems like the empty string used to be handled as "no valid devices exist" but changed meaning, as it is not mentioned in the documentation.

Changing the code to os.environ["CUDA_VISIBLE_DEVICES"] = "-1" fixes the problem. Running

import os
os.environ["CUDA_VISIBLE_DEVICES"]="-1"    
import tensorflow as tf

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

now prints

[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 14097726166554667970
]

and instantiating a tf.Session does not hog GPU memory anymore.