How does Python bypass normal attribute lookup to find `__dict__`?

Question

I understand that __dict__ in obj.__dict__ is a descriptor attribute of type(obj), so the lookup for obj.__dict__ is type(obj).__dict__['__dict__'].__get__(obj).

From https://stackoverflow.com/a/46576009

It's tempting to say that __dict__ has to be a descriptor because implementing it as a __dict__ entry would require you to find the __dict__ before you can find the __dict__, but Python already bypasses normal attribute lookup to find __dict__ when looking up other attributes, so that's not quite as compelling as it initially sounds. If the descriptors were replaced with a '__dict__' key in every __dict__, __dict__ would still be findable.

How does "Python already bypasses normal attribute lookup to find __dict__"? What does "normal attribute lookup" mean?

According to the context of the quote in the link, I don't think when the author wrote that, he referred to that the lookup for obj.__dict__ is type(obj).__dict__['__dict__'].__get__(obj).

Answer 1

Normal attribute lookup is done by calling the __getattribute__ hook, or more precisely, the C-API tp_getattro slot. The default implementation for this is in the PyObject_GenericGetAttr C-API function.

It is the job of PyObject_GenericGetAttr to invoke descriptors if they exist, and to look at the instance __dict__. And indeed, there is a __dict__ descriptor, but it is faster for __getattribute__ to just access the __dict__ slot in the instance memory structure directly, and that is what the actual implementation does:

if (dict == NULL) {
    /* Inline _PyObject_GetDictPtr */
    dictoffset = tp->tp_dictoffset;
    if (dictoffset != 0) {
        if (dictoffset < 0) {
            Py_ssize_t tsize;
            size_t size;

            tsize = ((PyVarObject *)obj)->ob_size;
            if (tsize < 0)
                tsize = -tsize;
            size = _PyObject_VAR_SIZE(tp, tsize);
            assert(size <= PY_SSIZE_T_MAX);

            dictoffset += (Py_ssize_t)size;
            assert(dictoffset > 0);
            assert(dictoffset % SIZEOF_VOID_P == 0);
        }
        dictptr = (PyObject **) ((char *)obj + dictoffset);
        dict = *dictptr;
    }
}

Note the Inline _PyObject_GetDictPtr comment; this is a performance optimisation, as instance attribute lookups are frequent.

If you try to access instance.__dict__ from Python code, then the descriptor is invoked; it is a data descriptor object so is invoked before instance attributes are even looked at.