Why I cannot override search path of dynamic libraries with LD_LIBRARY_PATH?

Question

Edit: I resolved this issue, the solution is below.

I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control fi

Answer 1

From http://man7.org/linux/man-pages/man8/ld.so.8.html

When resolving shared object dependencies, the dynamic linker first inspects each dependency string to see if it contains a slash (this can occur if a shared object pathname containing slashes was specified at link time). If a slash is found, then the dependency string is interpreted as a (relative or absolute) pathname, and the shared object is loaded using that pathname.

If a shared object dependency does not contain a slash, then it is searched for in the following order:

o (ELF only) Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist. Use of DT_RPATH is deprecated.

o Using the environment variable LD_LIBRARY_PATH. Except if the executable is a set-user-ID/set-group-ID binary, in which case it is ignored.

o (ELF only) Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present.

o From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented library path. If, however, the binary was linked with the -z nodeflib linker option, shared objects in the default paths are skipped. Shared objects installed in hardware capability directories (see below) are preferred to other shared objects.

o In the default path /lib, and then /usr/lib. (On some 64-bit archiectures, the default paths for 64-bit shared objects are /lib64, and then /usr/lib64.) If the binary was linked with the -z nodeflib linker option, this step is skipped.

• with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.

• with export LD_DEBUG=libs you can debug the search path used to find your libs

• with chrpath -r the rpath can be changed