Problems compiling Python 3.2 and 2.7 using pythonbrew

Question

I\'m trying to build multiple versions of python using Pythonbrew, but I\'m getting some test failures. This is on a VM running: Ubuntu 8.04 32bit

Answer 1

Currently, I manage to get pythonbrew working with 2.7.2 doing the following:

a) First, install all the dependencies needed to compile python.

'curl' # not for build, but for steps after
'build-essential',
'libbz2-dev',
'libsqlite3-dev',
'zlib1g-dev',
'libxml2-dev',
'libxslt1-dev',
'libreadline5', # lenny
'libreadline5-dev', # lenny
'libgdbm-dev',
'libgdb-dev',
'libxml2',
'libssl-dev',
'tk-dev',
'libgdbm-dev',
'libexpat1-dev',
'libncursesw5-dev'

b) Now install 2.7.2:

pythonbrew install --configure="--with-threads --enable-shared" -j2 -v 2.7.2

This installs, but is hit by the following bug:

/home/python-deploy/.pythonbrew/pythons/Python-2.7.2/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

*To avoid getting an error after the build when it tries to install setuptools, use the --no-setuptools option, and you might need to use --force on some platforms, as the gdb test fails due to an error in the test itself.

pythonbrew install --configure="--with-threads--enable-shared" \
                   --force \
                   --no-setuptools \
                   --jobs=2 \
                   --verbose 2.7.2 

c) The easiest workaround is to do (with the correct path to your pythonbrew install):

pythonbrew use 2.7.2
export LD_LIBRARY_PATH=$HOME/.pythonbrew/pythons/Python-2.7.2/lib

or to add it permanently, put the pythonbrew lib path in a file placed under /etc/ld.so.conf.d and the path will be added at startup, or right away running ldconfig:

sudo echo /home/user/.pythonbrew/pythons/Python-2.7.2/lib >> /etc/ld.so.conf.d/pythonbrew.conf
sudo ldconfig

d) Install setuptools or distribute manually, i.e.

curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py && easy_install pip

And you should be ready to go. Only tested with 2.7.2 on a vagrant debian lenny VM.

Answer 2

I think I found a way to make this work without having to set LD_LIBRARY_PATH.

I found out that there is an environment variable called LD_RUN_PATH that is just like LD_LIBRARY_PATH, except that you set it while compiling a program, and it remembers that path whenever it is run (so you don't need to set LD_LIBRARY_PATH at runtime).

So if you set LD_RUN_PATH appropriately while running pythonbrew to install your Python, it should find its own shared library whenever you run it. For example, the following command succeeded for me, including installing distutils and pip, so that module installation is ready to go without any further setup.

LD_RUN_PATH=$HOME/.pythonbrew/pythons/Python-2.7.3/lib pythonbrew install -j2 -C '--enable-shared' --force -v 2.7.3

Then I can do pythonbrew use 2.7.3 to activate it and it just works, without setting any environment variables in your shell init or anything.

Of course, you need to know in advance where pythonbrew will install your Python so that you can set LD_RUN_PATH correctly. Generally, it would be $HOME/.pythonbrew/pythons/Python-$VERSION/lib, but if that doesn't work you can always find out by actually installing it once (without --enable-shared) and then finding the lib directory.

I believe the errors occur because after installation with --enable-shared, your installed Python finds the system Python's shared library and uses that, which causes problems if there is any kind of mismatch in the way the two Pythons were built. Using either the LD_LIBRARY_PATH or the LD_RUN_PATH solution forces your Python to use its own shared library.

Answer 3

Ryan Thompson's answer is good ... LD_RUN_PATH is the proper remedy.

However, rather than hardcoding an absolute path, one can instead use:

LD_RUN_PATH='$ORIGIN/../lib'

... where $ORIGIN is the PWD of the running binary (eg. python).

# objdump -x /home/arisinger/.pythonbrew/pythons/Python-2.7.3/bin/python|grep RPATH
  RPATH                $ORIGIN/../lib
# ldd .pythonbrew/pythons/Python-2.7.3/bin/python
    [...]
    libpython2.7.so.1.0 => /home/arisinger/.pythonbrew/pythons/Python-2.7.3/bin/../lib/libpython2.7.so.1.0 (0x00007f43994f6000)
    [...]

... the above will also properly install pip, etc. Lastly, from man ld.so:

$ORIGIN and rpath
    ld.so  understands the string $ORIGIN (or equivalently ${ORIGIN}) in an
    rpath specification (DT_RPATH or DT_RUNPATH) to mean the directory con‐
    taining  the  application  executable.  Thus, an application located in
    somedir/app could be compiled with gcc  -Wl,-rpath,'$ORIGIN/../lib'  so
    that  it  finds  an  associated shared library in somedir/lib no matter
    where somedir is located in the directory hierarchy.  This  facilitates
    the  creation  of  "turn-key"  applications  that  do  not  need  to be
    installed into special directories, but can instead  be  unpacked  into
    any directory and still find their own shared libraries.

EDIT: sadly venvs breaks because virtualenv copies the python binary (changing $ORIGIN), but does not copy (or symlink) the library, so C/builtin dynamic modules in the venv end up linking to the system python ... ungood :-( ...

... IMO this is a bug to be fixed (in virtualenv?), but an easy workaround is to manually copy/symlink libpythonX.Y.so* into the lib/ dir of the venv ... or do something a little yucky looking like (note X.Y.Z):

LD_RUN_PATH='$ORIGIN/../lib:$ORIGIN/../../../../pythons/Python-X.Y.Z/lib'

... pick whatever is less nasty to you; both verified to work.