Why are explicit calls to magic methods slower than “sugared” syntax?

Question

I was messing around with a small custom data object that needs to be hashable, comparable, and fast, when I ran into an odd-looking set of timing results. Some of the comparisons (and the hashing method) for this object simply delegate to an attribute, so I was using something like:

def __hash__(self):
    return self.foo.__hash__()

However upon testing, I discovered that hash(self.foo) is noticeably faster. Curious, I tested __eq__, __ne__, and the other magic comparisons, only to discover that all of them ran faster if I used the sugary forms (==, !=, <, etc.). Why is this? I assumed the sugared form would have to make the same function call under the hood, but perhaps this isn't the case?

Timeit results

Setups: thin wrappers around an instance attribute that controls all the comparisons.

Python 3.3.4 (v3.3.4:7ff62415e426, Feb 10 2014, 18:13:51) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> 
>>> sugar_setup = '''\
... import datetime
... class Thin(object):
...     def __init__(self, f):
...             self._foo = f
...     def __hash__(self):
...             return hash(self._foo)
...     def __eq__(self, other):
...             return self._foo == other._foo
...     def __ne__(self, other):
...             return self._foo != other._foo
...     def __lt__(self, other):
...             return self._foo < other._foo
...     def __gt__(self, other):
...             return self._foo > other._foo
... '''
>>> explicit_setup = '''\
... import datetime
... class Thin(object):
...     def __init__(self, f):
...             self._foo = f
...     def __hash__(self):
...             return self._foo.__hash__()
...     def __eq__(self, other):
...             return self._foo.__eq__(other._foo)
...     def __ne__(self, other):
...             return self._foo.__ne__(other._foo)
...     def __lt__(self, other):
...             return self._foo.__lt__(other._foo)
...     def __gt__(self, other):
...             return self._foo.__gt__(other._foo)
... '''

Tests

My custom object is wrapping a datetime, so that's what I used, but it shouldn't make any difference. Yes, I'm creating the datetimes within the tests, so there's obviously some associated overhead there, but that overhead is constant from one test to another so it shouldn't make a difference. I've omitted the __ne__ and __gt__ tests for brevity, but those results were essentially identical to the ones shown here.

>>> test_hash = '''\
... for i in range(1, 1000):
...     hash(Thin(datetime.datetime.fromordinal(i)))
... '''
>>> test_eq = '''\
... for i in range(1, 1000):
...     a = Thin(datetime.datetime.fromordinal(i))
...     b = Thin(datetime.datetime.fromordinal(i+1))
...     a == a # True
...     a == b # False
... '''
>>> test_lt = '''\
... for i in range(1, 1000):
...     a = Thin(datetime.datetime.fromordinal(i))
...     b = Thin(datetime.datetime.fromordinal(i+1))
...     a < b # True
...     b < a # False
... '''

Results

>>> min(timeit.repeat(test_hash, explicit_setup, number=1000, repeat=20))
1.0805227295846862
>>> min(timeit.repeat(test_hash, sugar_setup, number=1000, repeat=20))
1.0135617737162192
>>> min(timeit.repeat(test_eq, explicit_setup, number=1000, repeat=20))
2.349765956168767
>>> min(timeit.repeat(test_eq, sugar_setup, number=1000, repeat=20))
2.1486044757355103
>>> min(timeit.repeat(test_lt, explicit_setup, number=500, repeat=20))
1.156479287717275
>>> min(timeit.repeat(test_lt, sugar_setup, number=500, repeat=20))
1.0673696685109917

• Hash:
  • Explicit: 1.0805227295846862
  • Sugared: 1.0135617737162192
• Equal:
  • Explicit: 2.349765956168767
  • Sugared: 2.1486044757355103
• Less Than:
  • Explicit: 1.156479287717275
  • Sugared: 1.0673696685109917

Answer 1

Two reasons:

• The API lookups look at the type only. They don't look at self.foo.__hash__, they look for type(self.foo).__hash__. That's one less dictionary to look in.

• The C slot lookup is faster than the pure-Python attribute lookup (which will use __getattribute__); instead looking up the method objects (including the descriptor binding) is done entirely in C, bypassing __getattribute__.

So you'd have to cache the type(self._foo).__hash__ lookup locally, and even then the call would not be as fast as from C code. Just stick to the standard library functions if speed is at a premium.

Another reason to avoid calling the magic methods directly is that the comparison operators do more than just call one magic method; the methods have reflected versions too; for x < y, if x.__lt__ isn't defined or x.__lt__(y) returns the NotImplemented singleton, y.__gt__(x) is consulted as well.