Can I patch a Python decorator before it wraps a function?

Question

I have a function with a decorator that I'm trying test with the help of the Python Mock library. I'd like to use mock.patch to replace the real decorator with a mock 'bypass' decorator which just calls the function. What I can't figure out is how to apply the patch before the real decorator wraps the function. I've tried a few different variations on the patch target and reordering the patch and import statements, but without success. Any ideas?

Answer 1

Decorators are applied at function definition time. For most functions, this is when the module is loaded. (Functions that are defined in other functions have the decorator applied each time the enclosing function is called.)

So if you want to monkey-patch a decorator, what you need to do is:

1. Import the module that contains it
2. Define the mock decorator function
3. Set e.g. module.decorator = mymockdecorator
4. Import the module(s) that use the decorator, or use it in your own module

If the module that contains the decorator also contains functions that use it, those are already decorated by the time you can see them, and you're probably S.O.L.

Edit to reflect changes to Python since I originally wrote this: If the decorator uses functools.wraps() and the version of Python is new enough, you may be able to dig out the original function using the __wrapped__ attritube and re-decorate it, but this is by no means guaranteed, and the decorator you want to replace also may not be the only decorator applied.

Answer 2

It should be noted that several of the answers here will patch the decorator for the entire test session rather than a single test instance; which may be undesirable. Here's how to patch a decorator that only persists through a single test.

Our unit to be tested with the undesired decorator:

# app/uut.py

from app.decorators import func_decor

@func_decor
def unit_to_be_tested():
    # Do stuff
    pass

From decorators module:

# app/decorators.py

def func_decor(func):
    def inner(*args, **kwargs):
        print "Do stuff we don't want in our test"
        return func(*args, **kwargs)
    return inner

By the time our test gets collected during a test run, the undesired decorator has already been applied to our unit under test (because that happens at import time). In order to get rid of that, we'll need to manually replace the decorator in the decorator's module and then re-import the module containing our UUT.

Our test module:

#  test_uut.py

from unittest import TestCase
from app import uut  # Module with our thing to test
from app import decorators  # Module with the decorator we need to replace
import imp  # Library to help us reload our UUT module
from mock import patch


class TestUUT(TestCase):
    def setUp(self):
        # Do cleanup first so it is ready if an exception is raised
        def kill_patches():  # Create a cleanup callback that undoes our patches
            patch.stopall()  # Stops all patches started with start()
            imp.reload(uut)  # Reload our UUT module which restores the original decorator
        self.addCleanup(kill_patches)  # We want to make sure this is run so we do this in addCleanup instead of tearDown

        # Now patch the decorator where the decorator is being imported from
        patch('app.decorators.func_decor', lambda x: x).start()  # The lambda makes our decorator into a pass-thru. Also, don't forget to call start()          
        # HINT: if you're patching a decor with params use something like:
        # lambda *x, **y: lambda f: f
        imp.reload(uut)  # Reloads the uut.py module which applies our patched decorator

The cleanup callback, kill_patches, restores the original decorator and re-applies it to the unit we were testing. This way, our patch only persists through a single test rather than the entire session -- which is exactly how any other patch should behave. Also, since the clean up calls patch.stopall(), we can start any other patches in the setUp() we need and they will get cleaned up all in one place.

The important thing to understand about this method is how the reloading will affect things. If a module takes too long or has logic that runs on import, you may just need to shrug and test the decorator as part of the unit. :( Hopefully your code is better written than that. Right?

If one doesn't care if the patch is applied to the whole test session, the easiest way to do that is right at the top of the test file:

# test_uut.py

from mock import patch
patch('app.decorators.func_decor', lambda x: x).start()  # MUST BE BEFORE THE UUT GETS IMPORTED ANYWHERE!

from app import uut

Make sure to patch the file with the decorator rather than the local scope of the UUT and to start the patch before importing the unit with the decorator.

Interestingly, even if the patch is stopped, all the files that already imported will still have the patch applied to the decorator, which is the reverse of the situation we started with. Be aware that this method will patch any other files in the test run that are imported afterwards -- even if they don't declare a patch themselves.

Answer 3

When I first ran across this problem, I use to rack my brain for hours. I found a much easier way to handle this.

This will fully bypass the decorator, like the target wasn't even decorated in the first place.

This is broken down into two parts. I suggest reading the following article.

http://alexmarandon.com/articles/python_mock_gotchas/

Two Gotchas that I kept running into:

1.) Mock the Decorator before the import of your function/module.

The decorators and functions are defined at the time the module is loaded. If you do not mock before import, it will disregard the mock. After load, you have to do a weird mock.patch.object, which gets even more frustrating.

2.) Make sure you are mocking the correct path to the decorator.

Remember that the patch of the decorator you are mocking is based on how your module loads the decorator, not how your test loads the decorator. This is why I suggest always using full paths for imports. This makes things a lot easier for testing.

Steps:

1.) The Mock function:

from functools import wraps

def mock_decorator(*args, **kwargs):
    def decorator(f):
        @wraps(f)
        def decorated_function(*args, **kwargs):
            return f(*args, **kwargs)
        return decorated_function
    return decorator

2.) Mocking the decorator:

2a.) Path inside with.

with mock.patch('path.to.my.decorator', mock_decorator):
     from mymodule import myfunction

2b.) Patch at top of file, or in TestCase.setUp

mock.patch('path.to.my.decorator', mock_decorator).start()

Either of these ways will allow you to import your function at anytime within the TestCase or its method/test cases.

from mymodule import myfunction

2.) Use a separate function as a side effect of the mock.patch.

Now you can use mock_decorator for each decorator you want to mock. You will have to mock each decorator separately, so watch out for the ones you miss.

Answer 4

The following worked for me:

1. Eliminate the import statement that loads the test target.
2. Patch the decorator on test startup as applied above.
3. Invoke importlib.import_module() immediately after patching to load the test target.
4. Run tests normally.

It worked like a charm.

Answer 5

Maybe you can apply another decorator onto the definitions of all your decorators that basically checks some configuration variable to see if testing mode is meant to be used.
If yes, it replaces the decorator it is decorating with a dummy decorator that does nothing.
Otherwise, it lets this decorator through.

Answer 6

Concept

This may sound a bit odd but one can patch sys.path, with a copy of itself, and perform an import within the scope of the test function. The following code shows the concept.

from unittest.mock import patch
import sys

@patch('sys.modules', sys.modules.copy())
def testImport():
 oldkeys = set(sys.modules.keys())
 import MODULE
 newkeys = set(sys.modules.keys())
 print((newkeys)-(oldkeys))

oldkeys = set(sys.modules.keys())
testImport()                       -> ("MODULE") # Set contains MODULE
newkeys = set(sys.modules.keys())
print((newkeys)-(oldkeys))         -> set()      # An empty set

MODULE may then be substituted with the module you are testing. (This works in Python 3.6 with MODULE substituted with xml for example)

OP

For your case, let's say the decorator function resides in the module pretty and the decorated function resides in present, then you would patch pretty.decorator using the mock machinery and substitute MODULE with present. Something like the following should work (Untested).

class TestDecorator(unittest.TestCase) : ...

  @patch(`pretty.decorator`, decorator)
  @patch(`sys.path`, sys.path.copy())
  def testFunction(self, decorator) :
   import present
   ...

Explanation

This works by providing a "clean" sys.path for each test function, using a copy of the current sys.path of the test module. This copy is made when the module is first parsed ensuring a consistent sys.path for all the tests.

Nuances

There are a few implications, however. If the testing framework runs multiple test modules under the same python session any test module that imports MODULE globally breaks any test module that imports it locally. This forces one to perform the import locally everywhere. If the framework runs each test module under a separate python session then this should work. Similarly you may not import MODULE globally within a test module where you're importing MODULE locally.

The local imports must be done for each test function within a subclass of unittest.TestCase. It is perhaps possible to apply this to the unittest.TestCase subclass directly making a particular import of the module available for all of the test functions within the class.

Built Ins

Those messing with builtin imports will find replacing MODULE with sys, os etc. will fail, since these are alread on sys.path when you try to copy it. The trick here is to invoke Python with the builtin imports disabled, I think python -X test.py will do it but I forget the appropriate flag (See python --help). These may subsequently be imported locally using import builtins, IIRC.

Answer 7

for @lru_cache(max_size=1000)

class MockedLruCache(object):

def __init__(self, maxsize=0, timeout=0):
    pass

def __call__(self, func):
    return func


cache.LruCache = MockedLruCache

if use decorator which haven't params, you should:

def MockAuthenticated(func):
    return func

from tornado import web
web.authenticated = MockAuthenticated