SWIG interfacing C library to Python (Creating 'iterable' Python data type from C 'sequence' struct)

Question

I have written a Python extension for a C library. I have a data structure that looks like this:

typedef struct _mystruct{
   double * clientdata;
   size_t          


        

Answer 1

I encountered the very same problem with Python 2.6, and solved it thank to @aphex reply. But I wanted to avoid any magic value, or extra boolean to pass end-of-list condition. Sure enough, my iterator have an atEnd() methods that tells me I am past the end of the list.

So in fact, it is fairly easy with SWIG exception handling. I just had to add the following magic:

%ignore MyStructIter::atEnd();
%exception MyStructIter::next {
    if( $self->list->atEnd() ) {
        PyErr_SetString(PyExc_StopIteration,"End of list");
        SWIG_fail;
    }
    $action
}

The point is this snipet skips the next() calls completly once you are past the end of list.

If you stick to your idioms, it should look like:

%exception MyStructIter::next {
    if( $self->pos >= $self->list->len ) {
        PyErr_SetString(PyExc_StopIteration,"End of list");
        SWIG_fail;
    }
    $action
}

NOTE FOR PYTHON 3.x:

You shall name your next() function with the magic "__ " prefix&postfix name. One option is simply to add:

%rename(__next__) MyStructIter::next;

Answer 2

The simplest solution to this is to implement __getitem__ and throw an IndexError exception for an invalid index.

I put together an example of this, using %extend and %exception in SWIG to implement __getitem__ and raise an exception respectively:

%module test

%include "exception.i"

%{
#include <assert.h>
#include "test.h"
static int myErr = 0; // flag to save error state
%}

%exception MyStruct::__getitem__ {
  assert(!myErr);
  $action
  if (myErr) {
    myErr = 0; // clear flag for next time
    // You could also check the value in $result, but it's a PyObject here
    SWIG_exception(SWIG_IndexError, "Index out of bounds");
  }
}

%include "test.h"

%extend MyStruct {
  double __getitem__(size_t i) {
    if (i >= $self->len) {
      myErr = 1;
      return 0;
    }
    return $self->clientdata[i];
  }
}

I tested it by adding to test.h:

static MyStruct *test() {
  static MyStruct inst = {0,0};
  if (!inst.clientdata) {
    inst.len = 10;
    inst.clientdata = malloc(sizeof(double)*inst.len);
    for (size_t i = 0; i < inst.len; ++i) {
      inst.clientdata[i] = i;
    }
  }
  return &inst;
}

And running the following Python:

import test

for i in test.test():
  print i

Which prints:

python run.py
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0

and then finishes.

---

An alternative approach, using a typemap to map MyStruct onto a PyList directly is possible too:

%module test

%{
#include "test.h"
%}

%typemap(out) (MyStruct *) {
  PyObject *list = PyList_New($1->len);
  for (size_t i = 0; i < $1->len; ++i) {
    PyList_SetItem(list, i, PyFloat_FromDouble($1->clientdata[i]));
  }

  $result = list;
}

%include "test.h"

This will create a PyList with the return value from any function that returns a MyStruct *. I tested this %typemap(out) with the exact same function as the previous method.

You can also write a corresponding %typemap(in) and %typemap(freearg) for the reverse, something like this untested code:

%typemap(in) (MyStruct *) {
  if (!PyList_Check($input)) {
    SWIG_exception(SWIG_TypeError, "Expecting a PyList");
    return NULL;
  }
  MyStruct *tmp = malloc(sizeof(MyStruct));
  tmp->len = PyList_Size($input);
  tmp->clientdata = malloc(sizeof(double) * tmp->len);
  for (size_t i = 0; i < tmp->len; ++i) {
    tmp->clientdata[i] = PyFloat_AsDouble(PyList_GetItem($input, i));
    if (PyErr_Occured()) {
      free(tmp->clientdata);
      free(tmp);
      SWIG_exception(SWIG_TypeError, "Expecting a double");
      return NULL;
    }
  }
  $1 = tmp;
}

%typemap(freearg) (MyStruct *) {
  free($1->clientdata);
  free($1);
}

---

Using an iterator would make more sense for containers like linked lists, but for completeness sake here's how you might go about doing it for MyStruct with __iter__. The key bit is that you get SWIG to wrap another type for you, which provides the __iter__() and next() needed, in this case MyStructIter which is defined and wrapped at the same time using %inline since it's not part of the normal C API:

%module test

%include "exception.i"

%{
#include <assert.h>
#include "test.h"
static int myErr = 0;
%}

%exception MyStructIter::next {
  assert(!myErr);
  $action
  if (myErr) {
    myErr = 0; // clear flag for next time
    PyErr_SetString(PyExc_StopIteration, "End of iterator");
    return NULL;
  }
}

%inline %{
  struct MyStructIter {
    double *ptr;
    size_t len;
  };
%}

%include "test.h"

%extend MyStructIter {
  struct MyStructIter *__iter__() {
    return $self;
  }

  double next() {
    if ($self->len--) {
      return *$self->ptr++;
    }
    myErr = 1;
    return 0;
  }
}

%extend MyStruct {
  struct MyStructIter __iter__() {
    struct MyStructIter ret = { $self->clientdata, $self->len };
    return ret;
  }
}

The requirements for iteration over containers are such that the container needs to implement __iter__() and return a new iterator, but in addition to next() which returns the next item and increments the iterator the iterator itself must also supply a __iter__() method. This means that either the container or an iterator can be used identically.

MyStructIter needs to keep track of the current state of iteration - where we are and how much we have left. In this example I did that by keeping a pointer to the next item and a counter that we use to tell when we hit the end. You could also have kept track of the sate by keeping a pointer to the MyStruct the iterator is using and a counter for the position within that, something like:

%inline %{
  struct MyStructIter {
    MyStruct *list;
    size_t pos;
  };
%}

%include "test.h"

%extend MyStructIter {
  struct MyStructIter *__iter__() {
    return $self;
  }

  double next() {
    if ($self->pos < $self->list->len) {
      return $self->list->clientdata[$self->pos++];
    }
    myErr = 1;
    return 0;
  }
}

%extend MyStruct {
  struct MyStructIter __iter__() {
    struct MyStructIter ret = { $self, 0 };
    return ret;
  }
}

(In this instance we could actually have just used the container itself as the iterator as an iterator, by supplying an __iter__() that returned a copy of the container and a next() similar to the first type. I didn't do that in my original answer because I thought that would be less clear than have two distinct types - a container and an iterator for that container)

Answer 3

1. Look up using the %typemap swig command. http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn25 The memberin typemap might do what you want. http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn35 I have a typemap that I found in the Python section that allows me to transfer char** data into the C++ as a list of Python strings. I would guess there would be similar functionality.
2. Also, you can define %pythoncode in your interface inside the struct inside the swig "i" file. This will allow you to add python methods in the object that gets created for the struct. There is another command %addmethod (I think) that allows you to add methods to the struct or a class as well. Then you can create methods for indexing the objects in C++ or C if you want. There are a lot of ways to solve this.

For an interface I am working on I used a class object that has some methods for accessing the data in my code. Those methods are written in C++. Then I used the %pythoncode directive inside the class inside of the "i" file and created "getitem" and "setitem" methods in Python code that uses the expose C++ methods to make it look like a dictionary style access.

Answer 4

You say you have yet to implement Python exception throwing - that's the problem. From PEP 234:

A new exception is defined, StopIteration, which can be used to signal the end of an iteration.

You must set this exception at the end of your iteration. Since your code doesn't do this, you're running into the situation you've described:

1. The interpreter loops through your list's custom iternext function
2. Your function gets to the end of the array, and rather than correctly setting the StopIteration exception, simply returns your 'magic number'.
3. The interpreter, seeing no good reason to stop iterating, simply continues to print the value returned by iternext... your magic number. To the interpreter, it's just yet another list member.

Fortunately, this is a pretty simple fix, though may not seem as straightforward, because C has no exception facility. The Python C API simply uses a global error indicator that you set when an exception situation is raised, and then the API standards dictate you return NULL all the way up the stack to the interpreter, which then looks at the output of PyErr_Occurred() to see if an error is set, and if it is, prints the relevant exception and traceback.

So in your function, when you reach the end of the array, you just need this:

PyErr_SetString(PyExc_StopIteration,"End of list");
return NULL;

Here's another great answer for further reading on this issue: How to create a generator/iterator with the Python C API?