Dealing with multiple Python versions and PIP?

Question

Is there any way to make pip play well with multiple versions of Python? For example, I want to use pip to explicitly install things to either my s

Answer 1

Here is my take on the problem. Works for Python3. The main features are:

• Each Python version is compiled from source
• All versions are installed locally
• Does not mangle your system's default Python installation in any way
• Each Python version is isolated with virtualenv

Prerequisites: If you are using some bare-bones thin client with no extra turf installed, you should run this first (in ubuntu 18.04 at least, extra packages added for convenience):

sudo apt-get update
sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update
sudo apt-get install -y build-essential cmake

sudo apt-get install -y zlib1g zlib1g-dev libsqlite3-dev \
openssl libssl-dev libffi-dev unzip pciutils net-tools \
libblas-dev gfortran libblas3 

The steps are as follows:

0. If you have several extra python versions installed in some other way, get rid of them, e.g., remove $HOME/.local/lib/python3.x, etc. (also the globally installed ones). Don't touch your system's default python3 version though.

1. Download source for different python versions under the following directory structure:

    $HOME/
        python_versions/ : download Python-*.tgz packages here and "tar xvf" them.  You'll get directories like this:
          Python-3.4.8/
          Python-3.6.5/
          Python-3.x.y/
          ...
   

2. At each "Python-3.x.y/" directory, do the following (do NOT use "sudo" in any of the steps!):

    mkdir root
    ./configure --prefix=$PWD/root 
    make -j 2
    make install
    virtualenv --no-site-packages -p root/bin/python3.x env
   

3. At "python_versions/" create files like this:

    env_python3x.bash:
   
    #!/bin/bash
    echo "type deactivate to exit"
    source $HOME/python_versions/Python-3.x.y/env/bin/activate
   

4. Now, anytime you wish to opt for python3.x, do

    source $HOME/python_versions/env_python3x.bash
   

to enter the virtualenv

6. While in the virtualenv, install your favorite python packages with

    pip install --upgrade package_name
   

7. To exit the virtualenv and python version just type "deactivate"

UPDATE

It seems that --no-site-packages is deprecated. There's an easy fix for this: Once you have activated the virtualenv, just point the HOME env variable to somewhere else than your actual home directory, i.e.:

export HOME=some/where/else

A nice way to do this in general is:

• Create virtualenv
• Activate virtualenv
• If you want to "recycle" existing libraries to your virtualenv, softlink them from your existing install, i.e. ln -s $HOME/.local/lib/python3.6/site-packages/numpy $PWD/venv/lib/python3.6/site-packages/
• Do export PYTHONPATH=, export HOME=/some/other/dir

Now you should have custom-isolated virtualenv.

UPDATE 2 / SUDO

Wan't to force sudo to use your virtualenv?

Defaults        secure_path="/home/USENAME/Python-3.x.y/env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
Defaults        env_keep += "VIRTUAL_ENV"
Defaults        env_keep += "PYTHONPATH"

Now try "sudo python3 --version" and magic should happen

UPDATE 3 / DOCKER

Enable virtualenv inside your docker (of course, you have built it in your docker image):

ENV VIRTUAL_ENV=/home/USER/Python-3.x.y/env
ENV PYTHONPATH=
ENV PATH="$VIRTUAL_ENV/bin:$PATH"