Python abc module: Extending both an abstract base class and an exception-derived class leads to surprising behavior

Question

Extending both an abstract base class and a class derived from "object" works as you would expect: if you you haven't implemented all abstract methods and properties, you get an error.

Strangely, replacing the object-derived class with an class that extends "Exception" allows you to create instances of classes which do not implement all the required abstract methods and properties.

For example:

import abc

# The superclasses
class myABC( object ):
    __metaclass__ = abc.ABCMeta

    @abc.abstractproperty
    def foo(self):
        pass

class myCustomException( Exception ):
    pass

class myObjectDerivedClass( object ):
    pass

# Mix them in different ways
class myConcreteClass_1(myCustomException, myABC):
    pass

class myConcreteClass_2(myObjectDerivedClass, myABC):
    pass

# Get surprising results
if __name__=='__main__':
    a = myConcreteClass_1()
    print "First instantiation done. We shouldn't get this far, but we do."
    b = myConcreteClass_2()
    print "Second instantiation done. We never reach here, which is good."

...yields...

First instantiation done. We shouldn't get this far, but we do.
Traceback (most recent call last):
  File "C:/Users/grahamf/PycharmProjects/mss/Modules/mssdevice/sutter/sutter/test.py", line 28, in <module>
    b = myConcreteClass_2()
TypeError: Can't instantiate abstract class myConcreteClass_2 with abstract methods foo

I know that "Exception" and therefore "myCustomException" have no attribute "foo", so why am I getting away with instantiating "myCustomException"?

EDIT: For the record, this is the hackish workaround I ended up going with. Not truly equivalent, but works for my purposes.

# "abstract" base class
class MyBaseClass( Exception ):
    def __init__(self):
        if not hasattr(self, 'foo'):
            raise NotImplementedError("Please implement abstract property foo")


class MyConcreteClass( MyBaseClass ):
    pass

if __name__=='__main__':
    a = MyConcreteClass()
    print "We never reach here, which is good."

Answer 1

It looks like this is because the __new__ method for BaseException doesn't care about abstract methods/properties.

When you try to instantiate myConcreteClass_1, it ends up calling __new__ from the Exception class. When want to instantiate myConcreteClass_2, it calls the __new__ from object:

>>> what.myConcreteClass_1.__new__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: exceptions.Exception.__new__(): not enough arguments
>>> what.myConcreteClass_2.__new__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object.__new__(): not enough arguments

The Exception class doesn't provide a __new__ method, but it's parent, BaseException, does:

static PyObject *
BaseException_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    PyBaseExceptionObject *self;

    self = (PyBaseExceptionObject *)type->tp_alloc(type, 0);
    if (!self)
        return NULL;
    /* the dict is created on the fly in PyObject_GenericSetAttr */
    self->dict = NULL;
    self->traceback = self->cause = self->context = NULL;
    self->suppress_context = 0;

    if (args) {
        self->args = args;
        Py_INCREF(args);
        return (PyObject *)self;
    }

    self->args = PyTuple_New(0);
    if (!self->args) {
        Py_DECREF(self);
        return NULL;
    }

    return (PyObject *)self;
}

Compare this to the __new__ implementation for object:

static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    if (excess_args(args, kwds) &&
        (type->tp_init == object_init || type->tp_new != object_new)) {
        PyErr_SetString(PyExc_TypeError, "object() takes no parameters");
        return NULL;
    }

    if (type->tp_flags & Py_TPFLAGS_IS_ABSTRACT) {
        PyObject *abstract_methods = NULL;
        PyObject *builtins;
        PyObject *sorted;
        PyObject *sorted_methods = NULL;
        PyObject *joined = NULL;
        PyObject *comma;
        _Py_static_string(comma_id, ", ");
        _Py_IDENTIFIER(sorted);

        /* Compute ", ".join(sorted(type.__abstractmethods__))
           into joined. */
        abstract_methods = type_abstractmethods(type, NULL);
        if (abstract_methods == NULL)
            goto error;
        builtins = PyEval_GetBuiltins();
        if (builtins == NULL)
            goto error;
        sorted = _PyDict_GetItemId(builtins, &PyId_sorted);
        if (sorted == NULL)
            goto error;
        sorted_methods = PyObject_CallFunctionObjArgs(sorted,
                                                      abstract_methods,
                                                      NULL);
        if (sorted_methods == NULL)
            goto error;
        comma = _PyUnicode_FromId(&comma_id);
        if (comma == NULL)
            goto error;
        joined = PyUnicode_Join(comma, sorted_methods);
        if (joined == NULL)
            goto error;

        PyErr_Format(PyExc_TypeError,
                     "Can't instantiate abstract class %s "
                     "with abstract methods %U",
                     type->tp_name,
                     joined);
    error:
        Py_XDECREF(joined);
        Py_XDECREF(sorted_methods);
        Py_XDECREF(abstract_methods);
        return NULL;
    }
    return type->tp_alloc(type, 0);
}

As you can see object.__new__ has code to throw an error when there are abstract methods that aren't overridden, but BaseException.__new__ does not.

Answer 2

Dano's answer is accurate but is missing a workaround. You can reproduce the object code in your own __new__ method:

import abc, traceback

# The superclasses
class MyABC(abc.ABC):

    @property
    @abc.abstractmethod
    def foo(self):
        pass

class MyCustomException( Exception ):
    pass

class MyObjectDerivedClass( object ):
    pass

# Mix them in different ways
class MyConcreteClass_1(MyCustomException, MyABC):

    def __new__(cls, *args, **kwargs):
        ''' Same abstract checks than in object.__new__ '''
        res = super().__new__(cls, *args, **kwargs)
        if cls.__abstractmethods__:
            raise TypeError(f"Can't instantiate abstract class {cls.__name__} with abstract methods {','.join(sorted(cls.__abstractmethods__))}")
        return res

class MyConcreteClass_2(MyObjectDerivedClass, MyABC):
    pass

# No longer get surprising results
if __name__=='__main__':
    try:
        a = MyConcreteClass_1()
    except TypeError:
        traceback.print_exc()
    try:
        b = MyConcreteClass_2()
    except TypeError:
        traceback.print_exc()

Which yields the two expected exceptions.