Is it possible to use tox with conda-based Python installations?

问题

The Python testing tool tox seems to be designed to work with virtualenv. Can it also work on conda/anaconda-based Python installations?

回答1:

While tox can't make use of conda, you can use conda to "install" different Python versions where tox can find them (like it would find "normal" Python installations in those folders). The following is tested on Windows:

You need virtualenv installed via pip in the root conda environment. I suspect this is the virtualenv that will be used by tox. (I had to install virtualenv using pip install virtualenv to get the virtualenv command to work, even though conda list showed it as installed.)
Install the Python versions you want to test. This is easily done using conda create. tox will autodetect Python binaries on Windows in C:\python27, C:\python33, etc., so create environments using conda create -p C:\python27 python=2.7 etc.

回答2:

Yes, you need the conda version of virtualenv installed for this to work.

Try:

>conda install virtualenv

virtualenv                15.1.0                   py36_

Change to project directory containing tox.ini

>tox

回答3:

I made tox and conda work together in Windows by:

Installing virtualenv with conda in the environment which I use tox:

conda install virtualenv
Creating "Directory Junction" symlinks from C:\PythonXY to my actual environment path. This gets around the InterpreterNotFound-error:

mklink /J C:\PythonXY C:\real\path\to\myPythonXYenv

I have installed Anaconda in E:\Anaconda3\, and all my environments in E:\Anaconda3\envs\, e.g. E:\Anaconda3\envs\py27\

(See below for a script to make this quick and easy.)

Step 1 - Create environments with conda:

E:\dev> conda create -n py27 python=2.7 --yes
E:\dev> conda create -n py33 python=3.3 --yes
...
E:\dev> conda create -n py36 python=3.6 --yes

Step 2 - Create all the symlinks:

E:\dev> mklink /J C:\Python27 E:\Anaconda3\envs\py27
E:\dev> mklink /J C:\Python33 E:\Anaconda3\envs\py33
...
E:\dev> mklink /J C:\Python36 E:\Anaconda3\envs\py36

Note: I call conda create from a directory on the E-drive, so the --prefix/-p option is not required in order to install new environments in E:\Anaconda3\envs\.

An Easier Way:

Instead of going through the cumbersome process of setting this up for each environment/python version, one can use the ToxEnvMatcher-class added further down in this manner:

my_envs = os.path.join('E:\\', 'Anaconda3', 'envs')
tem = ToxEnvMatcher(my_envs)
for version in '27,34,35,36'.split(','):
    tem.make(version)

Edit: To make the script easier to use, I've added a new section to the file, (here assumed to be tox_with_conda.py,) so it can be called from cmd.exe:

C:\dev> python tox_with_conda.py E:\Anaconda3\envs 27 34 35 36 37

Edit 2: Can also be installed with pip: pip install tox_with_conda and used as:

C:\dev> python -m tox_with_conda E:\Anaconda3\envs 27 34 35 36 37

I'm using Python 3.6.3 and tox 2.9.1, but I don't know when/if earlier versions work too.

Defence: I assume that to some, this seems like a too cumbersome process (it's really not, though), or to much of a hack. But keep in mind that being able to use Anaconda/conda also reduces waste of time spent trying to install libraries, packages, ++++.

Please note:

I use tox with pytest, and have not noticed any impacts on my tests.
My tests are simple, and there is a chance I just haven't been exposed to issues yet.
Assumable, there are things I haven't thought of that might be relevant for others.

The class (also available here):

from subprocess import run
from os.path import join

DEFAULT_BASE = join('C:\\', 'Python')


class ToxEnvMatcher:
    """
    Utility to make conda environments work with tox.

    Conda envs might be in other locations than where `tox <https://tox.readthedocs.io>`_ expects them to be.

    A symbolic link 'Directory Junction' is created from expected location to the actual location.
    Intended for Windows to get around the ``InterpreterNotFound``-error.

    E.g.: tox expects to find Python 2.7 in ``C:\Python27``,
    but may actually be installed in another drive and location.

    Examples of use:

    .. code-block:: python

        my_envs = join('E:\\', 'Anaconda3', 'envs')
        tem = ToxEnvMatcher(my_envs)
        for version in '27,34,35,36'.split(','):
            tem.make(version)

    The class is utilized through ``argsparse`` so it can also be used from cmd.exe.

    Examples of use of th of using ``ToxEnvMatcher`` from cmd.exe:

    .. code-block:: none

        E:\dev> tox_with_conda.py E:\Anaconda3\envs 27 34 35 36 37

    It's possible to use the ``-b``/``--base`` option to override the default base location (``C:\Python``):

    .. code-block:: none

        E:\dev> tox_with_conda.py E:\Anaconda3\envs 27 34 35 36 37 --base D:\Python

    :param str envs_dir: The path to where new conda environments will be created
    :param str default_base: The base of the 'default' location. Usually it's ``C:\Python``
    """
    def __init__(self, envs_dir, default_base=DEFAULT_BASE):
        self.envs_dir = envs_dir
        self.default_base = default_base

    def __repr__(self):
        return '{}({})'.format(self.__class__.__name__, self.envs_dir)

    def make(self, version):
        """
        Take version and create conda environment with symlink from 'default tox location'.

        E.g.: given version='27' and environment folder ``{self.envs_dir}``:

         - ``conda create -p {self.envs_dir}\py27 python=2.7``
         - ``mklink /J C:\Python27 {self.envs_dir}\py27``

        :param str version: A string on the form 'XY', e.g. '27' or '36'
        :return: None
        :rtype: NoneType
        """
        if len(version) != 2 or not int(version):
            raise ValueError("Parameter 'version' must be on the form 'XY', and not '{}'".format(version))
        conda_cmd = self._create_cmd_args(version)
        symlink_cmd = self._create_symlink_args(version)
        run(conda_cmd, shell=True)
        run(symlink_cmd, shell=True)

    def _get_env_folder(self, version):
        return join(self.envs_dir, 'py{}'.format(version))

    def _create_cmd_args(self, version):
        env_dir = self._get_env_folder(version)
        python_version = '.'.join(version)
        conda_create = 'conda create -p {} python={} --yes'.format(env_dir, python_version)
        return conda_create.split(' ')

    def _create_symlink_args(self, version):
        env_dir = self._get_env_folder(version)
        return 'mklink /J {}{} {}'.format(self.default_base, version, env_dir).split(' ')

The added code for making it work from cmd is:

if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("env_dir",
                        help="The folder where conda environments should be installed.")
    parser.add_argument("versions", nargs='*',
                        help="The list of versions, formatted 'XY' where X is major and Y minor. E.g. '27 35 36'")
    parser.add_argument("-b", "--base", default=DEFAULT_BASE,
                        help="Base of the path which tox expects to find Python installed. "
                             "Default: {}.".format(DEFAULT_BASE))
    args = parser.parse_args()

    print('env_dir: ', args.env_dir)
    print('versions: ', args.versions)
    print('--base: ', args.base)

    tem = ToxEnvMatcher(args.env_dir, default_base=args.base)
    for version in args.versions:
        tem.make(version)

回答4:

I don't know how developed it is, but you can look at https://github.com/hayd/ctox.

回答5:

The tox-conda plugin should close that gap nowadays, but needs contributors who actively use conda to test and improve it.

from the README:

tox-conda is a plugin that provides integration with the conda package and environment manager for the tox automation tool. It's like having your cake and eating it, too!

By default, tox creates isolated environments using [virtualenv](https://virtualenv.pypa.io] and installs dependencies from pip.

In contrast, when using the tox-conda plugin tox will use conda to create environments, and will install specified dependencies from conda. This is useful for developers who rely on conda for environment management and package distribution but want to take advantage of the features provided by tox for test automation.

To install that plugin it needs to be installed alongside tox in the same virutal environment. To create a virtual environment containing tox and tox-conda this should suffice:

$ python3 -m venv toxbase
$ toxbase/bin/pip install tox tox-conda
[...]
Successfully installed tox-3.13.2 tox-conda-0.2.0
$ toxbase/bin/tox --version
3.13.1 imported from /home/ob/tmp/toxbase/lib/python3.6/site-packages/tox/__init__.py
registered plugins:
    tox-conda-0.2.0 at /home/ob/tmp/toxbase/lib/python3.6/site-packages/tox_conda/plugin.py

from then on tox can be used as a command line tool and kept current by upgrading it in the toxbase virtualenv. Another, more automated way would be to use pipx

来源：https://stackoverflow.com/questions/30555943/is-it-possible-to-use-tox-with-conda-based-python-installations

标签

python

conda

tox