interfacing Python and Torch7(Lua) via shared library

Question

I am trying to pass data (arrays) between python and lua and I want to manipulate the data in lua using the Torch7 framework. I figured this can best be done through C, sin

Answer 1

On Linux Lua modules don't link to the Lua library directly but instead expect to find the Lua API functions already loaded. This is usually done by exporting them from the interpreter using the -Wl,-E linker flag. This flag only works for symbols in executables, not shared libraries. For shared libraries there exists something similar: the RTLD_GLOBAL flag for the dlopen function. By default all shared libraries listed on the compiler command line are loaded using RTLD_LOCAL instead, but fortunately Linux reuses already opened library handles. So you can either:

Preload the Lua(JIT) library using RTLD_GLOBAL before it gets loaded automatically (which happens when you load libcluaf.so):

from ctypes import byref, cdll, c_int
import ctypes

lualib = ctypes.CDLL("libluajit-5.1.so", mode=ctypes.RTLD_GLOBAL)
l = cdll.LoadLibrary('absolute_path_to_so/libcluaf.so')
# ...

Or change the flags of the Lua(JIT) library handle afterwards using the RTLD_NOLOAD flag for dlopen. This flag is not in POSIX though, and you probably have to use C to do so. See e.g. here.

Answer 2

For exchanging data between python/numpy and lua/torch, you could try a library named "lutorpy". It does exactly what you are trying to do, share the memory and only pass the pointer with "asNumpyArray()" method.

import lutorpy as lua
import numpy as np

## run lua code in python with minimal modification:  replace ":" to "._"
t = torch.DoubleTensor(10,3)
print(t._size()) # the corresponding lua version is t:size()

## convert torch tensor to numpy array
### Note: the underlying object are sharing the same memory, so the conversion is instant
arr = t.asNumpyArray()
print(arr.shape)

## or, you can convert numpy array to torch tensor
xn = np.random.randn(100)
## convert the numpy array into torch tensor
xt = torch.fromNumpyArray(xn)

Answer 3

The lua_gettop is a function defined in the Lua .so, which in your case must be luajit.so. Looks like you link your lib to it, that's good, and then you link main to your lib, so presumably the c compiler finds the Lua functions used by main in the luajit. So far so good.

Now when you load your lib via ctypes in python, does luajit lib automatically get loaded? You would expect it to, but you should confirm, maybe you have to tell ctypes to load linked libs. Another possibility is that ctypes or the lib loader does not find luajit, perhaps because it looks in places where luajit is not located. To be sure you might want to try putting all the libs in same folder from where you call Python.

If that doesn't help, try a variant of what you tried: don't load your module in python, just load luajit directly using ctypes, and try calling some of its methods.