Dynamically created method and decorator, got error 'functools.partial' object has no attribute '__module__'

Question

I am currently using EndpointsModel to create a RESTful API for all my models on AppEngine. Since it is RESTful, these api have a lot of repeat code which I want to avoid.

Answer 1

I stumbled upon this and thought would mention my workaround for this.

As rightly mentioned by @samy-vilar python3 doesn't have this issue. I have some code that uses functools.wrap and needs to run on python2 as well as python3.

For python2 we use functools32 which is backport of python3's functools for python2. wraps implementation of this package works perfect. Additionally it provides lru_cache which is available only in python3 functools.

import sys 

if sys.version[0] == '2':
   from functools32 import wraps
else:
   from functools import wraps

Answer 2

I also stumbled upon this, I was really surprised, for me the issue was that partial objects are missing certain attributes, specifically __module__ and __name__

Being that wraps by default uses functools.WRAPPER_ASSIGNMENTS to update attributes, which defaults to ('__module__', '__name__', '__doc__') in python 2.7.6 anyway, there are a couple ways of dealing with this ...

Update only present attributes ...

import functools
import itertools

def wraps_safely(obj, attr_names=functools.WRAPPER_ASSIGNMENTS):
    return wraps(obj, assigned=itertools.ifilter(functools.partial(hasattr, obj), attr_names))

>>> def foo():
...     """ Ubiquitous foo function ...."""
... 
>>> functools.wraps(partial(foo))(foo)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'functools.partial' object has no attribute '__module__'
>>> wraps_safely(partial(foo))(foo)()
>>> 

Here we simply filter out all those attribute which aren't present.

Another approach would be to strictly deal with partial objects only, you could fold wraps with singledispatch and create wrapped partial objects whose attributes would be taken from the deepest func attribute.

Something along the lines:

import functools

def wraps_partial(wrapper, *args, **kwargs):
    """ Creates a callable object whose attributes will be set from the partials nested func attribute ..."""
    wrapper = wrapper.func
    while isinstance(wrapper, functools.partial):
        wrapper = wrapper.func
    return functools.wraps(wrapper, *args, **kwargs)

def foo():
    """ Foo function.
    :return: None """
    pass

>>> wraps_partial(partial(partial(foo)))(lambda : None).__doc__
' Foo Function, returns None '
>>> wraps_partial(partial(partial(foo)))(lambda : None).__name__
'foo'
>>> wraps_partial(partial(partial(foo)))(lambda : None)()
>>> pfoo = partial(partial(foo))
>>> @wraps_partial(pfoo)
... def not_foo():
...     """ Not Foo function ... """
... 
>>> not_foo.__doc__
' Foo Function, returns None '
>>> not_foo.__name__
'foo'
>>>

This is slightly better since now we can get the original functions docs which before defaulted to using the partial objects doc string.

This can be modified to only search if the current partial object doesn't already have the set attribute, which should be slightly faster when nesting many partial objects ...

UPDATE

It seems that python(CPython) 3 (at least 3.4.3) doesn't have this issue, since I don't know nor should I assume all versions of python 3 or other implementations such as Jython also share this issue here is another future ready approach

from functools import wraps, partial, WRAPPER_ASSIGNMENTS

try:
    wraps(partial(wraps))(wraps)
except AttributeError:
    @wraps(wraps)
    def wraps(obj, attr_names=WRAPPER_ASSIGNMENTS, wraps=wraps):
        return wraps(obj, assigned=(name for name in attr_names if hasattr(obj, name))) 

a couple things to note:

• we define a new wraps function only if we fail to wrap a partial, in case future versions of python2 or other versions fix this issue.
• we use the original wraps to copy the docs and other info
• we don't use ifilter since it was removed in python3, I've timeit with and without ifilter but the results where inconclusive, at least in python (CPython) 2.7.6, the difference was marginal at best either way...

Answer 3

This issue is fixed as of Python 2.7.11 (not sure which specific release it was fixed in). You can do functools.wraps on a functools.partial object in 2.7.11.

Answer 4

In Python 3.5 I have found that a reference to the original function is maintained in the partial. You can access it as .func:

from functools import partial

def a(b):
    print(b)


In[20]:  c=partial(a,5)

In[21]:  c.func.__module__
Out[21]: '__main__'

In[22]:  c.func.__name__
Out[22]: 'a'

Answer 5

If it is the case that it is caused by an issue with "wraps" in functools, there is nothing stopping you from writing your own partial that does not call wraps. According to the python documentation, this is a valid implementation of partial:

def partial(func, *args, **keywords):
    def newfunc(*fargs, **fkeywords):
        newkeywords = keywords.copy()
        newkeywords.update(fkeywords)
        return func(*(args + fargs), **newkeywords)
    newfunc.func = func
    newfunc.args = args
    newfunc.keywords = keywords
    return newfunc

Answer 6

In our case I solved this by subclassing functools.partial:

class WrappablePartial(functools.partial):

    @property
    def __module__(self):
        return self.func.__module__

    @property
    def __name__(self):
        return "functools.partial({}, *{}, **{})".format(
            self.func.__name__,
            self.args,
            self.keywords
        )

    @property
    def __doc__(self):
        return self.func.__doc__

NB you could also make use of __getattr__ to redirect queries, but I figured that was actually less readable (and makes it more difficult to insert any useful meta-data as with __name__)