I've tried building tensorflow from source as described in the installation guide. I've had success building it with cpu-only support and with the SIMD instruction sets, but I've run into trouble trying to build with CUDA support.
System information:
- Mint 18 Sarah
- 4.4.0-21-generic
- gcc 5.4.0
- clang 3.8.0
- Python 3.6.1
- Nvidia GeForece GTX 1060 6GB (Compute capability 6.1)
- CUDA 8.0.61
- CuDNN 6.0
Here's my attempt at building with CUDA, gcc, and SIMD:
kevin@yeti-mint ~/src/tensorflow $ bazel clean INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes. kevin@yeti-mint ~/src/tensorflow $ ./configure You have bazel 0.5.2 installed. Please specify the location of python. [Default is /home/kevin/.pyenv/shims/python]: Found possible Python library paths: /home/kevin/.pyenv/versions/tensorflow/lib/python3.6/site-packages Please input the desired Python library path to use. Default is [/home/kevin/.pyenv/versions/tensorflow/lib/python3.6/site-packages] /home/kevin/.pyenv/versions/3.6.1/lib/python3.6 Do you wish to build TensorFlow with MKL support? [y/N] No MKL support will be enabled for TensorFlow Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: Do you wish to use jemalloc as the malloc implementation? [Y/n] jemalloc enabled Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] No Google Cloud Platform support will be enabled for TensorFlow Do you wish to build TensorFlow with Hadoop File System support? [y/N] No Hadoop File System support will be enabled for TensorFlow Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] No XLA support will be enabled for TensorFlow Do you wish to build TensorFlow with VERBS support? [y/N] No VERBS support will be enabled for TensorFlow Do you wish to build TensorFlow with OpenCL support? [y/N] No OpenCL support will be enabled for TensorFlow Do you wish to build TensorFlow with CUDA support? [y/N] y CUDA support will be enabled for TensorFlow Do you want to use clang as CUDA compiler? [y/N] nvcc will be used as CUDA compiler Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 8.0]: Please specify the location where CUDA toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 6.0]: Please specify the location where cuDNN library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify a list of comma-separated Cuda compute capabilities you want to build with. You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Please note that each additional compute capability significantly increases your build time and binary size. [Default is: "6.1"]: Do you wish to build TensorFlow with MPI support? [y/N] MPI support will not be enabled for TensorFlow Configuration finished kevin@yeti-mint ~/src/tensorflow $ bazel build --config=opt --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --verbose_failures //tensorflow/tools/pip_package:build_pip_package WARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead. WARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead. INFO: Found 1 target... ERROR: /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/external/protobuf/BUILD:244:1: C++ compilation of rule '@protobuf//:js_embed' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command (cd /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/execroot/org_tensorflow && \ exec env - \ PATH=/home/kevin/.pyenv/shims:/home/kevin/.pyenv/shims:/home/kevin/.pyenv/bin:/home/kevin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/kevin/.local/bin \ PWD=/proc/self/cwd \ external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++11' -g0 -MD -MF bazel-out/host/bin/external/protobuf/_objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.d '-frandom-seed=bazel-out/host/bin/external/protobuf/_objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.o' -iquote external/protobuf -iquote bazel-out/host/genfiles/external/protobuf -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/protobuf/src/google/protobuf/compiler/js/embed.cc -o bazel-out/host/bin/external/protobuf/_objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 2. python: can't open file 'external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc': [Errno 2] No such file or directory Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 5.578s, Critical Path: 0.06s Turning off all extra flags:
kevin@yeti-mint ~/src/tensorflow $ bazel build --config=opt --verbose_failures //tensorflow/tools/pip_package:build_pip_packageWARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead. WARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead. INFO: Found 1 target... ERROR: /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/external/fft2d/BUILD.bazel:21:1: C++ compilation of rule '@fft2d//:fft2d' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command (cd /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/execroot/org_tensorflow && \ exec env - \ PATH=/home/kevin/.pyenv/shims:/home/kevin/.pyenv/shims:/home/kevin/.pyenv/bin:/home/kevin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/kevin/.local/bin \ PWD=/proc/self/cwd \ external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 -MD -MF bazel-out/host/bin/external/fft2d/_objs/fft2d/external/fft2d/fft/fftsg.d -iquote external/fft2d -iquote bazel-out/host/genfiles/external/fft2d -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/fft2d/fft/fftsg.c -o bazel-out/host/bin/external/fft2d/_objs/fft2d/external/fft2d/fft/fftsg.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 2. python: can't open file 'external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc': [Errno 2] No such file or directory Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 3.522s, Critical Path: 2.42s Trying with clang instead:
kevin@yeti-mint ~/src/tensorflow $ ./configure You have bazel 0.5.2 installed. Please specify the location of python. [Default is /home/kevin/.pyenv/shims/python]: Found possible Python library paths: /home/kevin/.pyenv/versions/tensorflow/lib/python3.6/site-packages Please input the desired Python library path to use. Default is [/home/kevin/.pyenv/versions/tensorflow/lib/python3.6/site-packages] /home/kevin/.pyenv/versions/3.6.1/lib/python3.6 Do you wish to build TensorFlow with MKL support? [y/N] No MKL support will be enabled for TensorFlow Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: Do you wish to use jemalloc as the malloc implementation? [Y/n] jemalloc enabled Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] No Google Cloud Platform support will be enabled for TensorFlow Do you wish to build TensorFlow with Hadoop File System support? [y/N] No Hadoop File System support will be enabled for TensorFlow Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] No XLA support will be enabled for TensorFlow Do you wish to build TensorFlow with VERBS support? [y/N] No VERBS support will be enabled for TensorFlow Do you wish to build TensorFlow with OpenCL support? [y/N] No OpenCL support will be enabled for TensorFlow Do you wish to build TensorFlow with CUDA support? [y/N] y CUDA support will be enabled for TensorFlow Do you want to use clang as CUDA compiler? [y/N] y Clang will be used as CUDA compiler Please specify which clang should be used as device and host compiler. [Default is /usr/bin/clang]: Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 8.0]: Please specify the location where CUDA toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 6.0]: Please specify the location where cuDNN library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify a list of comma-separated Cuda compute capabilities you want to build with. You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Please note that each additional compute capability significantly increases your build time and binary size. [Default is: "6.1"]: Do you wish to build TensorFlow with MPI support? [y/N] MPI support will not be enabled for TensorFlow Configuration finished kevin@yeti-mint ~/src/tensorflow $ bazel build --config=opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-msse4.2 --verbose_failures //tensorflow/tools/pip_package:build_pip_package WARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead. WARNING: /home/kevin/src/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead. INFO: Found 1 target... ~1300 lines of build warnings and info... ERROR: /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/external/nccl_archive/BUILD:33:1: C++ compilation of rule '@nccl_archive//:nccl' failed: clang failed: error executing command (cd /home/kevin/.cache/bazel/_bazel_kevin/b937ae7b9a1087aeb7862ab37155238c/execroot/org_tensorflow && \ exec env - \ CLANG_CUDA_COMPILER_PATH=/usr/bin/clang \ CUDA_TOOLKIT_PATH=/usr/local/cuda \ CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \ PWD=/proc/self/cwd \ PYTHON_BIN_PATH=/home/kevin/.pyenv/shims/python \ PYTHON_LIB_PATH=/home/kevin/.pyenv/versions/3.6.1/lib/python3.6 \ TF_CUDA_CLANG=1 \ TF_CUDA_COMPUTE_CAPABILITIES=6.1 \ TF_CUDA_VERSION=8.0 \ TF_CUDNN_VERSION=6 \ TF_NEED_CUDA=1 \ TF_NEED_OPENCL=0 \ /usr/bin/clang '-march=native' -mavx -mavx2 -mfma -msse4.2 '-march=native' -MD -MF bazel-out/local_linux-py3-opt/bin/external/nccl_archive/_objs/nccl/external/nccl_archive/src/reduce.cu.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/external/nccl_archive/_objs/nccl/external/nccl_archive/src/reduce.cu.pic.o' -iquote external/nccl_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/nccl_archive -iquote external/local_config_cuda -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -isystem external/local_config_cuda/cuda -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda/include -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c++11' -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '-DCUDA_MAJOR=0' '-DCUDA_MINOR=0' '-DNCCL_MAJOR=0' '-DNCCL_MINOR=0' '-DNCCL_PATCH=0' -Iexternal/nccl_archive/src -O3 -x cuda '-DGOOGLE_CUDA=1' '--cuda-gpu-arch=sm_61' -c bazel-out/local_linux-py3-opt/genfiles/external/nccl_archive/src/reduce.cu.cc -o bazel-out/local_linux-py3-opt/bin/external/nccl_archive/_objs/nccl/external/nccl_archive/src/reduce.cu.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. clang: error: Unsupported CUDA gpu architecture: sm_61 Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 25.030s, Critical Path: 12.66s This is consistent behavior on the current master branch (31aa360), r1.2 (5d8c0a6), and r1.1 (8ddd727). I've seen many github issues (8790, 9651, 10367) and a stack overflow post or two (here, I tried using gcc/g++ 4.8), but they all seem to be solved and/or slightly unrelated to my problem.