Test for import of optional dependencies in __init__.py with pytest: Python 3.5 /3.6 differs in behaviour

问题

I have a package for python 3.5 and 3.6 that has optional dependencies for which I want tests (pytest) that run on either version.

I made a reduced example below consisting of two files, a simple __init__.py where the optional package "requests" (just an example) is imported and a flag is set to indicate the availability of requests.

mypackage/
├── mypackage
│   └── __init__.py
└── test_init.py

The __init__.py file content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

requests_available = True

try:
    import requests
except ImportError:
    requests_available = False

The test_init.py file content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pytest, sys

def test_requests_missing(monkeypatch):
    import mypackage
    import copy
    fakesysmodules = copy.copy(sys.modules)
    fakesysmodules["requests"] = None
    monkeypatch.delitem(sys.modules,"requests")
    monkeypatch.setattr("sys.modules", fakesysmodules)
    from importlib import reload
    reload(mypackage)
    assert mypackage.requests_available == False


if __name__ == '__main__':
    pytest.main([__file__, "-vv", "-s"])

The test_requests_missing test works on Python 3.6.5:

runfile('/home/bjorn/python_packages/mypackage/test_init.py', wdir='/home/bjorn/python_packages/mypackage')
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.6.1, py-1.5.2, pluggy-0.6.0 -- /home/bjorn/anaconda3/envs/bjorn36/bin/python
cachedir: .pytest_cache
rootdir: /home/bjorn/python_packages/mypackage, inifile:
plugins: requests-mock-1.5.0, mock-1.10.0, cov-2.5.1, nbval-0.9.0, hypothesis-3.38.5
collecting ... collected 1 item

test_init.py::test_requests_missing PASSED

=========================== 1 passed in 0.02 seconds ===========================

But not on Python 3.5.4:

runfile('/home/bjorn/python_packages/mypackage/test_init.py', wdir='/home/bjorn/python_packages/mypackage')
========================================================= test session starts ==========================================================
platform linux -- Python 3.5.4, pytest-3.6.1, py-1.5.2, pluggy-0.6.0 -- /home/bjorn/anaconda3/envs/bjorn35/bin/python
cachedir: .pytest_cache
rootdir: /home/bjorn/python_packages/mypackage, inifile:
plugins: requests-mock-1.5.0, mock-1.10.0, cov-2.5.1, nbval-0.9.1, hypothesis-3.38.5
collecting ... collected 1 item

test_init.py::test_requests_missing FAILED

=============================================================== FAILURES ===============================================================
________________________________________________________ test_requests_missing _________________________________________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f9a2953acc0>

    def test_requests_missing(monkeypatch):
        import mypackage
        import copy
        fakesysmodules = copy.copy(sys.modules)
        fakesysmodules["requests"] = None
        monkeypatch.delitem(sys.modules,"requests")
        monkeypatch.setattr("sys.modules", fakesysmodules)
        from importlib import reload
>       reload(mypackage)

test_init.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../anaconda3/envs/bjorn35/lib/python3.5/importlib/__init__.py:166: in reload
    _bootstrap._exec(spec, module)
<frozen importlib._bootstrap>:626: in _exec
    ???
<frozen importlib._bootstrap_external>:697: in exec_module
    ???
<frozen importlib._bootstrap>:222: in _call_with_frames_removed
    ???
mypackage/__init__.py:8: in <module>
    import requests
../../anaconda3/envs/bjorn35/lib/python3.5/site-packages/requests/__init__.py:97: in <module>
    from . import utils

.... VERY LONG OUTPUT ....

    from . import utils
../../anaconda3/envs/bjorn35/lib/python3.5/site-packages/requests/__init__.py:97: in <module>
    from . import utils
<frozen importlib._bootstrap>:968: in _find_and_load
    ???
<frozen importlib._bootstrap>:953: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:896: in _find_spec
    ???
<frozen importlib._bootstrap_external>:1171: in find_spec
    ???
<frozen importlib._bootstrap_external>:1145: in _get_spec
    ???
<frozen importlib._bootstrap_external>:1273: in find_spec
    ???
<frozen importlib._bootstrap_external>:1245: in _get_spec
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'requests', location = '/home/bjorn/anaconda3/envs/bjorn35/lib/python3.5/site-packages/requests/__init__.py'

>   ???
E   RecursionError: maximum recursion depth exceeded

<frozen importlib._bootstrap_external>:575: RecursionError
======================================================= 1 failed in 2.01 seconds =======================================================

I have two questions:

Why do I see this difference? Relevant packages seem to be of the same version on both 3.5 and 3.6.
Is there a better way to do what I want? The code I have now is stitched together from examples found online. I have tried to patch the import mechanism in an attempt to avoid "reload", but I have not managed.

回答1:

I would mock the __import__ function (the one invoked behind the import modname statement). Example:

import builtins
import sys
import pytest


@pytest.fixture
def no_requests(monkeypatch):
    import_orig = builtins.__import__
    def mocked_import(name, globals, locals, fromlist, level):
        if name == 'requests':
            raise ImportError()
        return import_orig(name, locals, fromlist, level)
    monkeypatch.setattr(builtins, '__import__', mocked_import)


@pytest.fixture(autouse=True)
def cleanup_imports():
    yield
    sys.modules.pop('mypackage', None)


def test_requests_available():
    import mypackage
    assert mypackage.requests_available


@pytest.mark.usefixtures('no_requests')
def test_requests_missing():
    import mypackage
    assert not mypackage.requests_available

The fixture no_requests is responsible for replacing the __import__ function with one that will raise on import requests attempt, doing fine on the rest of imports (we can't raise on any import or pytest itself will break). cleanup_imports is just to ensure that mypackage will be reimported in each test.

回答2:

If a test tests optional functionality, it should be skipped rather than passed if that functionality is missing.

test.support.import_module() is the function used in the Python autotest suite to skip a test or a test file if a module is missing:

import test.support
import unittest

nonexistent = test.support.import_module("nonexistent")

class TestDummy(unittest.testCase):
    def test_dummy():
        self.assertTrue(nonexistent.vaporware())

Then, when running:

> python -m py.test -rs t.py

<...>
collected 0 items / 1 skipped

=========================== short test summary info ===========================
SKIP [1] C:\Python27\lib\test\support\__init__.py:90: SkipTest: No module named
nonexistent
========================== 1 skipped in 0.05 seconds ==========================

来源：https://stackoverflow.com/questions/51044068/test-for-import-of-optional-dependencies-in-init-py-with-pytest-python-3-5

标签

python

python-3.5

python-3.6

pytest