Why doesn't the abc.ABCMeta abstract instantiation check work on derivatives of `list` and `dict`?

Question

I have been experimenting a little with the abc module in python. A la

>>> import abc

In the normal case you expect your ABC class to not be instantiated if it contains an unimplemented abstractmethod. You know like as follows:

>>> class MyClass(metaclass=abc.ABCMeta):
...     @abc.abstractmethod
...     def mymethod(self):
...         return -1
...
>>> MyClass()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyClass with abstract methods mymethod

OR for any derived Class. It all seems to work fine until you inherit from something ... say dict or list as in the following:

>>> class YourClass(list, metaclass=abc.ABCMeta):
...     @abc.abstractmethod
...     def yourmethod(self):
...         return -1
...
>>> YourClass()
[]

This is surprising because type is probably the primary factory or metaclass -ish thing anyway or so I assume from the following.

>>> type(abc.ABCMeta)
<class 'type'>
>>> type(list)
<class 'type'>

From some investigation I found out that it boils down to something as simple as adding an __abstractmethod__ attribute to the class' object and rest happens by itself:

>>> class AbstractClass:
...     pass
...
>>> AbstractClass.__abstractmethods__ = {'abstractmethod'}
>>> AbstractClass()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class AbstractClass with abstract methods abstractmethod

So one can simply avoid the check by intentionally overriding the __new__ method and clearing out __abstractmethods__ as in below:

>>> class SupposedlyAbstractClass(metaclass=abc.ABCMeta):
...     def __new__(cls):
...         cls.__abstractmethods__ = {}
...         return super(AbstractClass, cls).__new__(cls)
...     @abc.abstractmethod
...     def abstractmethod(self):
...         return -1
...
>>> SupposedlyAbstractClass()
<__main__.SupposedlyAbstractClass object at 0x000001FA6BF05828>

This behaviour is the same in Python 2.7 and in Python 3.7 as I have personally checked. I am not aware if this is the same for all other python implementations.

Finally, down to the question ... Why has this been made to behave like so? Is it wise we should never make abstract classes out of list, tuple or dict? or should I just go ahead and add a __new__ class method checking for __abstractmethods__ before instantiation?

Answer 1

The problem

If you have the next class:

from abc import ABC, abstractmethod
class Foo(list, ABC):
    @abstractmethod
    def yourmethod(self):
        pass

the problem is that and object of Foo can be created without any error because Foo.__new__(Foo) delegates the call directly to list.__new__(Foo) instead of ABC.__new__(Foo) (which is responsible of checking that all abstract methods are implemented in the class that is going to be instantiated)

We could implement __new__ on Foo and try to call ABC.__new__:

class Foo(list, ABC):
    def __new__(cls, *args, **kwargs):
        return ABC.__new__(cls)

    @abstractmethod
    def yourmethod(self):
        pass
Foo()

But he next error is raised:

TypeError: object.__new__(Foo) is not safe, use list.__new__()

This is due to ABC.__new__(Foo) invokes object.__new__(Foo) which seems that is not allowed when Foo inherits from list

A possible solution

You can add additional code on Foo.__new__ in order to check that all abstract methods in the class to be instantiated are implemented (basically do the job of ABC.__new__).

Something like this:

class Foo(list, ABC):
    def __new__(cls, *args, **kwargs):
        if hasattr(cls, '__abstractmethods__') and len(cls.__abstractmethods__) > 0:
            raise TypeError(f"Can't instantiate abstract class {cls.__name__} with abstract methods {', '.join(cls.__abstractmethods__)}")
        return super(Foo, cls).__new__(cls)


    @abstractmethod
    def yourmethod(self):
        return -1

Now Foo() raises an error. But the next code runs without any issue:

class Bar(Foo):
     def yourmethod(self):
         pass
Bar()