Expose a non-const but noncopyable member in Boost Python

Question

Here\'s my problem:

I have two classes like these:

class Signal {
public:
    void connect(...) { sig.connect(.         


        

Answer 1

I am having issues reproducing your results, but here is some information that may help in resolving the problem.

With simple classes:

class Signal
{
public:
  void connect() { std::cout << "connect called" << std::endl; }
private:
  boost::signals2::signal<void()> signal_;
};

class MyClass
{
public:
  Signal on_event;
};

And basic bindings:

namespace python = boost::python;
python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect", &Signal::connect)
  ;

python::class_<MyClass>("MyClass")
  .def_readonly("on_event", &MyClass::on_event)
  ;

The code fails to compile. When exposing a class, Boost.Python's default behavior to register converters. These converters require copy constructors, as a means to copy C++ class object into storage that can be managed by a Python object. This behavior can be disabled for a class by providing boost::noncopyable as an argument to the class_ type.

In this instance, MyClass binding does not suppress copy constructors. Boost.Python will attempt to use copy constructors within the bindings, and fail with a compiler error, because the member variable on_event is not copyable. Signal is not copyable because its contains a member variable with a type of boost::signal2::signal, that inherits from boost::noncopyable.

Adding boost:::noncopyable as a argument type to MyClass's bindings allows for the code to compile.

namespace python = boost::python;
python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect", &Signal::connect)
  ;

python::class_<MyClass, boost::noncopyable>("MyClass")
  .def_readonly("on_event", &MyClass::on_event)
  ;

Usage:

>>> import example
>>> m = example.MyClass()
>>> m.on_event.connect()
connect called
>>> m.on_event = None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: can't set attribute
>>> 

While this setup allows for the desired bindings and calling syntax, it looks as though it is the first step in the final goal.

---

My apologies if this is too presumptuous. However, based on other recent questions, I would like to take the time to expand upon the initial example to cover what appears to be the final goal: being able to connect Python callbacks to a signal2::signal. I will cover two different approaches, as the mechanics and level of complexity differ, but they may provide insight into details that should be considered for the final solution.

Only Python threads.

For this first scenario, lets assume that only Python threads are interacting with the library.

One technique that keeps it relatively simple is to use inheritance. Start by defining a helper Slot class that can connect to Signal.

class Slot
  : public boost::python::wrapper<Slot>
{
public:
  void operator()()
  {
    this->get_override("__call__")();
  }
};

The Slot class inherits from boost::python::wrapper, a class that unintrusively provides hooks to allow Python classes to override functions in the base class.

When a callable type connects to boost::signals2::signal, the signal may copy the argument into its internal list. Thus, it is important for the functor to be able to extend the life of the Slot instance for as long as it remains connected to signal. The easiest way to accomplish this is by managing Slot via a boost::shared_ptr.

The resulting Signal class looks like:

class Signal
{
public:
  template <typename Callback>
  void connect(const Callback& callback)
  {
    signal_.connect(callback);
  }

  void operator()() { signal_(); }
private:
  boost::signals2::signal<void()> signal_;
};

And a helper function helps keep Signal::connect generic, in case other C++ types need to connect to it.

void connect_slot(Signal& self, 
                  const boost::shared_ptr<Slot>& slot)
{
  self.connect(boost::bind(&Slot::operator(), slot));
}

This results in the following bindings:

BOOST_PYTHON_MODULE(example) {
  namespace python = boost::python;
  python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
    .def("connect",  &connect_slot)
    .def("__call__", &Signal::operator())
    ;

  python::class_<MyClass, boost::noncopyable>("MyClass")
    .def_readonly("on_event", &MyClass::on_event)
    ;

  python::class_<Slot, boost::shared_ptr<Slot>, 
                 boost::noncopyable>("Slot")
    .def("__call__", python::pure_virtual(&Slot::operator()))
    ;
}

And its usage is as follows:

>>> from example import *
>>> class Foo(Slot):
...     def __call__(self):
...          print "Foo::__call__"
... 
>>> m = MyClass()
>>> foo = Foo()
>>> m.on_event.connect(foo)
>>> m.on_event()
Foo::__call__
>>> foo = None
>>> m.on_event()
Foo::__call__

While successful, it has an unfortunate characteristic of not being pythonic. For example:

>>> def spam():
...     print "spam"
... 
>>> m = MyClass()
>>> m.on_event.connect(spam)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    Signal.connect(Signal, function)
did not match C++ signature:
    connect(Signal {lvalue}, boost::shared_ptr<Slot>)

It would be ideal if any callable object could be connected to the signal. One easy way to this it to monkey patch the bindings in Python. To be transparent to the end user:

• Change the C++ binding module name from example to _example. Make sure to also change the library name.
• Create example.py that will patch Signal.connect() to wrap the argument into a type that inherits from Slot.

example.py could look something like this:

from _example import *

class _SlotWrap(Slot):

    def __init__(self, fn):
        self.fn = fn
        Slot.__init__(self)

    def __call__(self):
        self.fn()

def _signal_connect(fn):
    def decorator(self, slot):
        # If the slot is not an instance of Slot, then aggregate it
        # in SlotWrap.
        if not isinstance(slot, Slot):
            slot = _SlotWrap(slot)
        # Invoke the decorated function with the slot.
        return fn(self, slot)
    return decorator

# Patch Signal.connect.
Signal.connect = _signal_connect(Signal.connect)

The patching is seamless to the end user.

>>> from example import *
>>> def spam():
...     print "spam"
... 
>>> m = MyClass()
>>> m.on_event.connect(spam)
>>> m.on_event()
spam

With this patch, any callable type can connect to Signal without having to explicitly inherit from Slot. As such, it becomes much more pythonic than the initial solution. Never underestimate the benefit in keeping the bindings simple and non-pythonic, but patch them to be pythonic in python.

---

Python and C++ threads.

In the next scenario, lets consider the case where C++ threads are interacting with Python. For example, a C++ thread can be set to invoke the signal after a period of time.

This example can become fairly involved, so lets start with the basics: Python's Global Interpreter Lock (GIL). In short, the GIL is a mutex around the interpreter. If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. In the previous example, as there were no C++ threads, all actions occurred while the GIL had been acquired. While this fairly straightforward, it can become complex rather quickly.

First, the module needs to have Python initialize the GIL for threading.

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.
  ...
}

For convenience, lets create a simple class to help manage the GIL:

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

The thread will be invoking MyClass's signal. Thus, it needs to extend the lifetime of MyClass while the thread is alive. A good candidate to accomplish this is by managing MyClass with a shared_ptr.

Lets identify when the C++ thread will need the GIL:

• MyClass being deleted by shared_ptr.
• boost::signals2::signal can make additional copies of connected objects, as is done when the signal is concurrently invoked.
• Invoking a Python objected connected through boost::signals2::signal. The callback will certainly affect python objects. For example, the self argument provided to the __call__ method will increase and decrease an object's reference count.

Supporting MyClass being deleted from a C++ thread.

To guarantee the GIL is held when MyClass is deleted by shared_ptr from within a C++ thread, a custom deleter is required. This also requires the bindings to suppress the default constructor, and use a custom constructor instead.

/// @brief Custom deleter.
template <typename T>
struct py_deleter
{
  void operator()(T* t)
  {
    gil_lock lock;    
    delete t;
  }
};

/// @brief Create Signal with a custom deleter.
boost::shared_ptr<MyClass> create_signal()
{
  return boost::shared_ptr<MyClass>(
    new MyClass(),
    py_deleter<MyClass>());
}

...

BOOST_PYTHON_MODULE(example) {

  ...

  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass", python::no_init)
    .def("__init__", python::make_constructor(&create_signal))
    .def_readonly("on_event", &MyClass::on_event)
    ;
}

The thread itself.

The thread's functionality is fairly basic: it sleeps then invokes the signal. However, it is important to understand the context of the GIL.

/// @brief Wait for a period of time, then invoke the
///        signal on MyClass.
void call_signal(boost::shared_ptr<MyClass>& shared_class,
                 unsigned int seconds)
{
  // The shared_ptr was created by the caller when the GIL was
  // locked, and is accepted as a reference to avoid modifying
  // it while the GIL is not locked.

  // Sleep without the GIL so that other python threads are able
  // to run.
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking C++-specific
  // slots connected to the signal.  Thus, it is the responsibility of
  // python slots to lock the GIL.  Additionally, the potential
  // copying of slots internally by the signal will be handled through
  // another mechanism.
  shared_class->on_event();

  // The shared_class has a custom deleter that will lock the GIL
  // when deletion needs to occur.
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
void spawn_signal_thread(boost::shared_ptr<MyClass> self,
                       unsigned int seconds)
{
  // The caller owns the GIL, so it is safe to make copies.  Thus,
  // spawn off the thread, binding the arguments via copies.  As
  // the thread will not be joined, detach from the thread.
  boost::thread(boost::bind(&call_signal, self, seconds)).detach();
}

And the MyClass bindings are updated.

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass", python::no_init)
  .def("__init__", python::make_constructor(&create_signal))
  .def("signal_in", &spawn_signal_thread)
  .def_readonly("on_event", &MyClass::on_event)
  ;

boost::signals2::signal interacting with python objects.

boost::signals2::signal may make copies when it is invoked. Additionally, there may be C++ slots connected to the signal, so it would be ideal to not have the GIL locked while the signal is invoked. However, signal does not provide hooks to allow us to acquire the GIL before creating copies of slots or invoking the slot.

To add to the complexity, when the bindings expose a C++ function that accepts a C++ class with a HeldType that is not a smart pointer, then Boost.Python will extract the non-reference counted C++ object from the reference-counted python object. It can safely do this because the calling thread, in Python, has the GIL. To maintain a reference count to slots trying to connect from Python, as well as allow for any callable type to connect, we can use the opaque type of boost::python::object.

In order to avoid having signal create copies of the provided boost::python::object, one can create a copy of boost::python::object so that reference counts remain accurate, and manage the copy via shared_ptr. This allows signal to freely create copies of shared_ptr instead of creating boost::python::object without the GIL.

This GIL safety slot can be encapsulated in a helper class.

/// @brief Helper type that will manage the GIL for a python slot.
class py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(new boost::python::object(object),   // GIL locked, so copy.
              py_deleter<boost::python::object>()) // Delete needs GIL.
  {}

  void operator()()
  {
    // Lock the gil as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

A helper function will be exposed to Python to help adapt the types.

/// @brief Signal connect helper.
void signal_connect(Signal& self,
                    boost::python::object object)
{
  self.connect(boost::bind(&py_slot::operator(), py_slot(object)));
}

And the updated binding expose the helper function:

python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
  .def("connect",  &signal_connect)
  .def("__call__", &Signal::operator())
  ;

The final solution looks like this:

#include <boost/bind.hpp>
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/signals2/signal.hpp>
#include <boost/thread.hpp>

class Signal
{
public:
  template <typename Callback>
  void connect(const Callback& callback)
  {
    signal_.connect(callback);
  }

  void operator()() { signal_(); }
private:
  boost::signals2::signal<void()> signal_;
};

class MyClass
{
public:
  Signal on_event;
};

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};    

/// @brief Custom deleter.
template <typename T>
struct py_deleter
{
  void operator()(T* t)
  {
    gil_lock lock;    
    delete t;
  }
};

/// @brief Create Signal with a custom deleter.
boost::shared_ptr<MyClass> create_signal()
{
  return boost::shared_ptr<MyClass>(
    new MyClass(),
    py_deleter<MyClass>());
}

/// @brief Wait for a period of time, then invoke the
///        signal on MyClass.
void call_signal(boost::shared_ptr<MyClass>& shared_class,
                 unsigned int seconds)
{
  // The shared_ptr was created by the caller when the GIL was
  // locked, and is accepted as a reference to avoid modifying
  // it while the GIL is not locked.

  // Sleep without the GIL so that other python threads are able
  // to run.
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking C++-specific
  // slots connected to the signal.  Thus, it is the responsibility of
  // python slots to lock the GIL.  Additionally, the potential
  // copying of slots internally by the signal will be handled through
  // another mechanism.
  shared_class->on_event();

  // The shared_class has a custom deleter that will lock the GIL
  // when deletion needs to occur.
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
void spawn_signal_thread(boost::shared_ptr<MyClass> self,
                       unsigned int seconds)
{
  // The caller owns the GIL, so it is safe to make copies.  Thus,
  // spawn off the thread, binding the arguments via copies.  As
  // the thread will not be joined, detach from the thread.
  boost::thread(boost::bind(&call_signal, self, seconds)).detach();
}

/// @brief Helepr type that will manage the GIL for a python slot.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(new boost::python::object(object),   // GIL locked, so copy.
              py_deleter<boost::python::object>()) // Delete needs GIL.
  {}

  void operator()()
  {
    // Lock the gil as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

/// @brief Signal connect helper.
void signal_connect(Signal& self,
                    boost::python::object object)
{
  self.connect(boost::bind(&py_slot::operator(), py_slot(object)));
}

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<Signal, boost::noncopyable>("Signal", python::no_init)
    .def("connect",  &signal_connect)
    .def("__call__", &Signal::operator())
    ;

  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass", python::no_init)
    .def("__init__", python::make_constructor(&create_signal))
    .def("signal_in", &spawn_signal_thread)
    .def_readonly("on_event", &MyClass::on_event)
    ;
}

And a testing script (test.py):

from time import sleep
from example import *

def spam():
    print "spam"

m = MyClass()
m.on_event.connect(spam)
m.on_event()

m.signal_in(2)
m = None
print "Sleeping"
sleep(5)
print "Done sleeping"

Results in the following:

spam
Sleeping
spam
Done sleeping

---

In conclusion, when an object is passed through the Boost.Python layer, take time to consider how to manage its lifespan and the context in which it will be used. This often requires understanding how other libraries being used will handle the object. It is not an easy problem, and providing a pythonic solution can be a challenge.

Answer 2

After writing this question I added a public copy constructor to Signal and now it works.