How to tidy/fix PyCXX's creation of new-style Python extension-class?

Question

I\'ve nearly finished rewriting a C++ Python wrapper (PyCXX).

The original allows old and new style extension classes, but also allows one to derive from the new-style c

Answer 1

Here is a small C example that shows how Python allocates memory for object of classes derived from C types:

typedef struct
{
    PyObject_HEAD
    int dummy[100];
} xxx_obj;

It also needs a type object:

static PyTypeObject xxx_type = 
{
    PyObject_HEAD_INIT(NULL)
};

And a module initialization function that initializes this type:

extern "C"
void init_xxx(void)
{
    PyObject* m;

    xxx_type.tp_name = "_xxx.xxx";
    xxx_type.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE;

    xxx_type.tp_new = tp_new; // IMPORTANT
    xxx_type.tp_basicsize = sizeof(xxx_obj); // IMPORTANT

    if (PyType_Ready(&xxx_type) < 0)
        return;

    m = Py_InitModule3("_xxx", NULL, "");

    Py_INCREF(&xxx_type);
    PyModule_AddObject(m, "xxx", (PyObject *)&xxx_type);
}

What is missing is the implementation of tp_new: The Python docs require that:

The tp_new function should call subtype->tp_alloc(subtype, nitems) to allocate space for the object

So lets do that and add a few printouts.

static
PyObject *tp_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds)
{
    printf("xxx.tp_new():\n\n");

    printf("\t subtype=%s\n", subtype->tp_name);
    printf("\t subtype->tp_base=%s\n", subtype->tp_base->tp_name);
    printf("\t subtype->tp_base->tp_base=%s\n", subtype->tp_base->tp_base->tp_name);

    printf("\n");

    printf("\t subtype->tp_basicsize=%ld\n", subtype->tp_basicsize);
    printf("\t subtype->tp_base->tp_basicsize=%ld\n", subtype->tp_base->tp_basicsize);
    printf("\t subtype->tp_base->tp_base->tp_basicsize=%ld\n", subtype->tp_base->tp_base->tp_basicsize);

    return subtype->tp_alloc(subtype, 0); // IMPORTANT: memory allocation is done here!
}

Now run a very simple Python program to test it. This program creates a new class derived from xxx, and then creates an object of type derived.

import _xxx

class derived(_xxx.xxx):
    def __init__(self):
        super(derived, self).__init__()

d = derived()

To create an object of type derived, Python will call its tp_new, which in turn will call its base class' (xxx) tp_new. This call generates the following output (exact numbers depends on the machine architecture):

xxx.tp_new():

    subtype=derived
    subtype->tp_base=_xxx.xxx
    subtype->tp_base->tp_base=object

    subtype->tp_basicsize=432
    subtype->tp_base->tp_basicsize=416
    subtype->tp_base->tp_base->tp_basicsize=16

The subtype argument to tp_new is the type of the object being created (derived), it derives from our C type (_xxx.xxx), which in turns derives from object. The base object is of size 16, which is just PyObject_HEAD, the xxx type has an additional 400 bytes for its dummy member for a total of 416 bytes and the derived Python class adds additional 16 bytes.

Because subtype->tp_basicsize accounts for the sizes of all three levels of the hierarchy (object, xxx and derived) for a total of 432 bytes, the right amount of memory is being allocated.