【转载】 SLI导致双显卡被TensorFlow同时占用问题（Windows下） ---------- (windows环境下如何为tensorflow安装多个独立的消费级显卡）

版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_21368481/article/details/81907244

————————————————

转载注明：
突然想给自己的电脑上tensorflow环境下安装多独立显卡，网上搜索发现这篇文章，该篇文章主要是在windows环境下为tensorflow安装多个独立显卡。

本文逻辑：
windows环境下安装多个独立显卡，如果不使用sli技术，则Windows不识别多个独立显卡，但是使用sli技术，则不能指定单独显卡为tensorflow进行计算，因为指定单独显卡后slave显卡的显存占用会和master显卡的显存占用进行同步，也就是即使指定了一个显卡参与运算但是另一个显卡的显存会随之同步变化，本文作者提出一个方法解决这个问题：Windows环境下两显卡进行物理桥接后在软件上关闭桥接功能，便可实现Windows环境下双显卡识别及单显卡指定运算。

原文如下：
---------------------------------------

最近学习TensorFlow，被一些不是bug的问题折腾的头晕脑胀，借此写一下解决方法。本人是在win10下使用TensorFlow的，所以ubuntu下的绕行吧，不会出现这些问题。（此文有些地方我重新整理了一遍，放在了相约机器人公众号上，大家可以参见链接）

众所周知，TensorFlow在运行时，会抢占所有检测到的GPU的显存，这种做法褒贬不一吧，只能说，但怎么单独设置使用哪几块显卡呢，唯一的方法就是利用CUDA本身隐藏掉某些显卡（除此之外就是拔掉多余显卡了，大家应该不会傻到这么去做），有些教辅书或网上教程中写的以下方法都是治标不治本的：

（1）使用with.....device语句

例如

with tf.device("/gpu:1"):

这只是指定下面的程序在哪块GPU上执行，程序本身还是会占用所有GPU的资源（信不信由你）

（2）使用allow_growth=True或per_process_gpu_memory_fraction

例如

import tensorflow as tf

g = tf.placeholder(tf.int16)
h = tf.placeholder(tf.int16)
mul = tf.multiply(g,h)
gpu_options = tf.GPUOptions(allow_growth = True)

# gpu_options = tf.GPUOptions ( per_process_gpu_memory_fraction = 0.7 )
config = tf.ConfigProto(log_device_placement = True,allow_soft_placement = True,gpu_options = gpu_options)
with tf.Session(config=config) as sess:
print("相乘：%d" % sess.run(mul, feed_dict = {g:3,h:4}))

前者能够实现随着程序本身慢慢增加所占用的GPU的显存，但仍旧会占用所有GPU，如下：

上图为程序运行前，下图为程序运行后，可见程序运行后，两块GPU均被占用了，但实际上只有GPU0执行了上述程序：

C:\Users\B622>python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>>
>>> g = tf.placeholder(tf.int16)
>>> h = tf.placeholder(tf.int16)
>>> mul = tf.multiply(g,h)
>>>
>>> gpu_options = tf.GPUOptions(allow_growth = True)
>>> #gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction = 0.7)
... config = tf.ConfigProto(log_device_placement = True,allow_soft_placement = True,gpu_options = gpu_options)
>>> with tf.Session(config=config) as sess:
...     print("相乘：%d" % sess.run(mul, feed_dict = {g:3,h:4}))
...
2018-08-21 07:00:01.651592: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-08-21 07:00:01.927932: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.721
pciBusID: 0000:17:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-08-21 07:00:02.025456: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.721
pciBusID: 0000:65:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-08-21 07:00:02.030441: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0, 1
2018-08-21 07:00:03.036953: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-21 07:00:03.040347: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]      0 1
2018-08-21 07:00:03.042564: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0:   N N
2018-08-21 07:00:03.044994: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 1:   N N
2018-08-21 07:00:03.047419: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8806 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)
2018-08-21 07:00:03.054450: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 8806 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1
/job:localhost/replica:0/task:0/device:GPU:1 -> device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1
2018-08-21 07:00:03.064623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\direct_session.cc:284] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1
/job:localhost/replica:0/task:0/device:GPU:1 -> device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0, compute capability: 6.1
 
Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0
2018-08-21 07:00:03.074668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:886] Mul: (Mul)/job:localhost/replica:0/task:0/device:GPU:0
Placeholder_1: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
2018-08-21 07:00:03.078028: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:886] Placeholder_1: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
Placeholder: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
2018-08-21 07:00:03.081462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:886] Placeholder: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
相乘：12

而后者设置固定大小资源的per_process_gpu_memory_fraction，也只是均匀抢占每块GPU这么多资源而已，仍旧占用了所有GPU，如下：

正确的做法是利用CUDA来隐藏某些GPU，方法如下：

（1）直接在代码中利用python语句实现

    import os
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
    os.environ["CUDA_VISIBLE_DEVICES"] = "1"

（2）直接在终端写入

windows下（test.py改成自己的py文件）：

set CUDA_VISIBLE_DEVICES=1
python tset.py

linux下：

CUDA_VISIBLE_DEVICES=1 python test.py

但是如果程序中出现with tf.device():等语句，可能会因为不小心的索引而发生错误，为什么这么说呢？

CUDA_VISIBLE_DEVICES=1           Only device 1 will be seen
CUDA_VISIBLE_DEVICES=0,1         Devices 0 and 1 will be visible
CUDA_VISIBLE_DEVICES="0,1"       Same as above, quotation marks are optional
CUDA_VISIBLE_DEVICES=0,2,3       Devices 0, 2, 3 will be visible; device 1 is masked

CUDA_VISIBLE_DEVICES=""          No GPU will be visible

举个例子，当运行如下代码时，程序会提示错误：

import tensorflow as tf
import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
 
with tf.device("/gpu:1"):
    g = tf.placeholder(tf.int16)
    h = tf.placeholder(tf.int16)
    mul = tf.multiply(g,h)
gpu_options = tf.GPUOptions(allow_growth = True)
config = tf.ConfigProto(log_device_placement = True,gpu_options = gpu_options)
#config = tf.ConfigProto(log_device_placement = True,allow_soft_placement = True)
with tf.Session(config=config) as sess:
    print("相乘：%d" % sess.run(mul, feed_dict = {g:3,h:4}))

因为当设置os.environ["CUDA_VISIBLE_DEVICES"] = "1"时，如果你又使用了with tf.device("/gpu:1"):（注：with tf.device("/gpu:0"):是正确的），则程序会提示你没有可用的GPU1，只有可用的CPU0和GPU0，如下（原因是因为设置了CUDA_VISIBLE_DEVICES后，CUDA本身会重新按你设置的顺序从0开始排列可见的GPU，这里只设置了一块GPU，所以只能索引到第0号GPU，超出索引会报错，虽然物理PCI总线上调用的还是GPU1这块显卡，但程序本身认为该块显卡的索引号是0而不是1）：

InvalidArgumentError: Cannot assign a device for operation 'Mul': Operation was explicitly assigned to /device:GPU:1 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0 ]. Make sure the device specification refers to a valid device.
     [[Node: Mul = Mul[T=DT_INT16, _device="/device:GPU:1"](Placeholder, Placeholder_1)]]
 
Caused by op 'Mul', defined at:
  File "E:\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 269, in <module>
    main()
  File "E:\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 265, in main
    kernel.start()
  File "E:\Anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 486, in start
    self.io_loop.start()
  File "E:\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 127, in start
    self.asyncio_loop.run_forever()
  File "E:\Anaconda3\lib\asyncio\base_events.py", line 422, in run_forever
    self._run_once()
  File "E:\Anaconda3\lib\asyncio\base_events.py", line 1432, in _run_once
    handle._run()
  File "E:\Anaconda3\lib\asyncio\events.py", line 145, in _run
    self._callback(*self._args)
  File "E:\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 117, in _handle_events
    handler_func(fileobj, events)
  File "E:\Anaconda3\lib\site-packages\tornado\stack_context.py", line 276, in null_wrapper
    return fn(*args, **kwargs)
  File "E:\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 450, in _handle_events
    self._handle_recv()
  File "E:\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 480, in _handle_recv
    self._run_callback(callback, msg)
  File "E:\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 432, in _run_callback
    callback(*args, **kwargs)
  File "E:\Anaconda3\lib\site-packages\tornado\stack_context.py", line 276, in null_wrapper
    return fn(*args, **kwargs)
  File "E:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "E:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "E:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "E:\Anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "E:\Anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "E:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2662, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "E:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2785, in _run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "E:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2909, in run_ast_nodes
    if self.run_code(code, result):
  File "E:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2963, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-f92d6fb2b710>", line 1, in <module>
    runfile('C:/Users/B622/.spyder-py3/temp.py', wdir='C:/Users/B622/.spyder-py3')
  File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)
  File "E:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)
  File "C:/Users/B622/.spyder-py3/temp.py", line 22, in <module>
    add2 = tf.multiply(g,h)
  File "E:\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 337, in multiply
    return gen_math_ops.mul(x, y, name)
  File "E:\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5066, in mul
    "Mul", x=x, y=y, name=name)
  File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
    op_def=op_def)
  File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
 
InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'Mul': Operation was explicitly assigned to /device:GPU:1 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0 ]. Make sure the device specification refers to a valid device.
     [[Node: Mul = Mul[T=DT_INT16, _device="/device:GPU:1"](Placeholder, Placeholder_1)]]

同理，如果你设置了os.environ["CUDA_VISIBLE_DEVICES"] = "3,0,1"（假设你有4块GPU），则这时物理上的GPU3在程序看来是GPU0，物理上的GPU0在程序看来是GPU1，物理上的GPU1在程序看来是GPU2，物理上的GPU2不可见（被隐藏掉了）。

当然为了防止不小心的索引，可以在tf.ConfigProto中设置allow_soft_placement = True（表示指定的设备不存在时，允许tf自动分配设备），但这其实和我们所要将某些代码指配给某块GPU相违背，所以在写tf.device时要想清楚现在的GPU索引号。

除上述之外，在windows下还有很坑的一点是，当你的机子上有两块GPU设置了交火后，即使用了SLI桥后，无论你怎么设置os.environ["CUDA_VISIBLE_DEVICES"] = "1"或在终端写入对应指定某块GPU的指令，TensorFlow还是会占用所有GPU，虽然真的只有设定的GPU可见。

是不是感觉隐藏的GPU不可用，但还是被占了显存，有点赔了夫人又折兵啊。就是这么荒唐，这个问题，排查了我一宿加一早上，百度又百度都找不到任何答案。尝试过拆除SLI桥（如下图）：