Manually incrementing and decrementing a boost::shared_ptr?

Question

Is there a way to manually increment and decrement the count of a shared_ptr in C++?

The problem that I am trying to solve is as follows. I am writing a library in C

Answer 1

In your suggestion

The client will then be responsible to decrement the counter.

means that the client in question is responsible for memory management, and that your trust her. I still do not understand why.

It is not possible to actually modify the shared_ptr counter... (hum, I'll explain at the end how to...) but there are other solutions.

Solution 1: complete ownership to the client

Hand over the pointer to the client (shared_ptr::release) and expect it to pass the ownership back to you when calling back (or simply deleting the object if it is not really shared).

That's actually the traditional approach when dealing with raw pointers and it apply here as well. The downside is that you actually release ownership for this shared_ptr only. If the object is actually shared that might prove inconvenient... so bear with me.

Solution 2: with a callback

This solution means that you always keep ownership and are responsible to maintain this object alive (and kicking) for as long as the client needs it. When the client is done with the object, you expect her to tell you so and invoke a callback in your code that will perform the necessary cleanup.

struct Object;

class Pool // may be a singleton, may be synchronized for multi-thread usage
{
public:
  int accept(boost::shared_ptr<Object>); // adds ptr to the map, returns NEW id
  void release(int id) { m_objects.erase(id); }

private:
  std::map< int, boost::shared_ptr<Object> > m_objects;
}; // class Pool

This way, your client 'decrementing' the counter is actually your client calling a callback method with the id you used, and you deleting one shared_ptr :)

Hacking boost::shared_ptr

As I said it is possible (since we are in C++) to actually hack into the shared_ptr. There are even several ways to do it.

The best way (and easiest) is simply to copy the file down under another name (my_shared_ptr ?) and then:

• change the include guards
• include the real shared_ptr at the beginning
• rename any instance of shared_ptr with your own name (and change the private to public to access the attributes)
• remove all the stuff that is already defined in the real file to avoid clashes

This way you easily obtain a shared_ptr of your own, for which you can access the count. It does not solve the problem of having the C code directly accessing the counter though, you may have to 'simplify' the code here to replace it by a built-in (which works if you are not multi-threaded, and is downright disastrous if you are).

I purposely left out the 'reinterpret_cast' trick and the pointer offsets ones. There are just so many ways to gain illegit access to something in C/C++!

May I advise you NOT to use the hacks though? The two solutions I presented above should be enough to tackle your problem.

Answer 2

Maybe you are using boost::shared_ptr accross DLL boundaries, what won't work properly. In this case boost::intrusive_ptr might help you out. This is a common case of misuse of shared_ptr people try to work around with dirty hacks... Maybe I am wrong in your case but there should be no good reason to do what you try to do ;-)

ADDED 07/2010: The issues seem to come more from DLL loading/unloading than from the shared_ptr itself. Even the boost rationale doesn't tell much about the cases when boost::intrusive_ptr should be preferred over shared_ptr. I switched to .NET development and didn't follow the details of TR1 regarding this topic, so beware this answer might not be valid anymore now...

Answer 3

fastest possible concurrent lockless manager (if you know what you are doing).

template< class T >
class shared_pool
{
public:

    typedef T value_type;
    typedef shared_ptr< value_type > value_ptr;
    typedef value_ptr* lock_handle;

shared_pool( size_t maxSize ):
    _poolStore( maxSize )
{}

// returns nullptr if there is no place in vector, which cannot be resized without locking due to concurrency
lock_handle try_acquire( const value_ptr& lockPtr ) {
    static value_ptr nullPtr( nullptr );
    for( auto& poolItem: _poolStore ) {
        if( std::atomic_compare_exchange_strong( &poolItem, &nullPtr, lockPtr ) ) {             
            return &poolItem;
        }
    }
    return nullptr;
}


lock_handle acquire( const value_ptr& lockPtr ) {
    lock_handle outID;
    while( ( outID = try_acquire( lockPtr ) ) == nullptr ) {
        mt::sheduler::yield_passive(); // ::SleepEx( 1, false );
    }
    return outID;
}

value_ptr release( const lock_handle& lockID ) {
    value_ptr lockPtr( nullptr );
    std::swap( *lockID, lockPtr);
    return lockPtr;
}

protected:

    vector< value_ptr > _poolStore;

};

std::map is not so fast, requires additional search, extra memory, spin-locking. But it grants extra safety with handles approach.

BTW, hack with manual release/acquire seems to be a much better approach (in terms of speed and memory usage). C++ std better add such a functionality in their classes, just to keep C++ razor-shaped.

Answer 4

Another option would be to just allocate dynamically a copy of the shared_ptr, in order to increment the refcount, and deallocate it in order to decrement it. This guarantees that my shared object will not be destroyed while in use by the C api client.

In the following code snippet, I use increment() and decrement() in order to control a shared_ptr. For the simplicity of this example, I store the initial shared_ptr in a global variable.

#include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/make_shared.hpp>
#include <boost/scoped_ptr.hpp>
using namespace std;

typedef boost::shared_ptr<int> MySharedPtr;
MySharedPtr ptr = boost::make_shared<int>(123);

void* increment()
{
    // copy constructor called
    return new MySharedPtr(ptr);
}

void decrement( void* x)
{
    boost::scoped_ptr< MySharedPtr > myPtr( reinterpret_cast< MySharedPtr* >(x) );
}

int main()
{
    cout << ptr.use_count() << endl;
    void* x = increment();
    cout << ptr.use_count() << endl;
    decrement(x);
    cout << ptr.use_count() << endl;

    return 0;
}

Output:

1
2
1

Answer 5

I came across a use case where I did need something like this, related to IOCompletionPorts and concurrency concerns. The hacky but standards compliant method is to lawyer it as described by Herb Sutter here.

The following code snippet is for std::shared_ptr as implemented by VC11:

Impl File:

namespace {
    struct HackClass {
        std::_Ref_count_base *_extracted;
    };
}

template<>
template<>
void std::_Ptr_base<[YourType]>::_Reset<HackClass>(std::auto_ptr<HackClass> &&h) {
     h->_extracted = _Rep; // Reference counter pointer
}

std::_Ref_count_base *get_ref_counter(const std::shared_ptr<[YourType]> &p) {
     HackClass hck;
     std::auto_ptr<HackClass> aHck(&hck);

     const_cast<std::shared_ptr<[YourType]>&>(p)._Reset(std::move(aHck));

     auto ret = hck._extracted; // The ref counter for the shared pointer
                                // passed in to the function

     aHck.release(); // We don't want the auto_ptr to call delete because
                     // the pointer that it is owning was initialized on the stack

     return ret;
}

void increment_shared_count(std::shared_ptr<[YourType]> &sp) {
     get_ref_counter(sp)->_Incref();
}

void decrement_shared_count(std::shared_ptr<[YourType]> &sp) {
     get_ref_counter(sp)->_Decref();
}

Replace [YourType] with the type of object you need to modify the count on. It is important to note that this is pretty hacky, and uses platform specific object names. The amount of work you have to go through to get this functionality is probably indicative of how bad of an idea it is. Also, I am playing games with the auto_ptr because the function I am hijacking from shared_ptr takes in an auto_ptr.

Answer 6

1. A handle?

If you want maximum security, gives the user a handle, not the pointer. This way, there's no way he will try to free it and half-succeed.

I'll assume below that, for simplicity's sake, you'll give the user the object pointer.

2. acquire and unacquire ?

You should create a manager class, as described by Matthieu M. in his answer, to memorize what was acquired/unacquired by the user.

As the inferface is C, you can't expect him to use delete or whatever. So, a header like:

#ifndef MY_STRUCT_H
#define MY_STRUCT_H

#ifdef __cplusplus
extern "C"
{
#endif // __cplusplus

typedef struct MyStructDef{} MyStruct ; // dummy declaration, to help
                                        // the compiler not mix types

MyStruct * MyStruct_new() ;
size_t     MyStruct_getSomeValue(MyStruct * p) ;
void       MyStruct_delete(MyStruct * p) ;

#ifdef __cplusplus
}
#endif // __cplusplus

#endif // MY_STRUCT_H

Will enable the user to use your class. I used a declaration of a dummy struct because I want to help the C user by not imposing him the use of the generic void * pointer. But using void * is still a good thing.

The C++ source implementing the feature would be:

#include "MyClass.hpp"
#include "MyStruct.h"

MyManager g_oManager ; // object managing the shared instances
                       // of your class

extern "C"
{

MyStruct * MyStruct_new()
{
   MyClass * pMyClass = g_oManager.createMyClass() ;
   MyStruct * pMyStruct = reinterpret_cast<MyStruct *>(pMyClass) ;
   return pMyStruct ;
}

size_t MyStruct_getSomeValue(MyStruct * p)
{
   MyClass * pMyClass = reinterpret_cast<MyClass *>(p) ;

   if(g_oManager.isMyClassExisting(pMyClass))
   {
      return pMyClass->getSomeValue() ;
   }
   else
   {
      // Oops... the user made a mistake
      // Handle it the way you want...
   }

   return 0 ;
}

void MyStruct_delete(MyStruct * p)
{
   MyClass * pMyClass = reinterpret_cast<MyClass *>(p) ;
   g_oManager.destroyMyClass(pMyClass) ;
}

}

Note that the pointer to MyStruct is plain invalid. You should not use it for whatever reason without reinterpret_cast-ing it into its original MyClass type (see Jaif's answer for more info on that. The C user will use it only with the associated MyStruct_* functions.

Note too that this code verify the class does exist. This could be overkill, but it is a possible use of a manager (see below)

3. About the manager

The manager will hold, as suggested by Matthieu M., a map containing the shared pointer as a value (and the pointer itself, or the handle, as the key). Or a multimap, if it is possible for the user to somehow acquire the same object multiple times.

The good thing about the use of a manager will be that your C++ code will be able to trace which objects were not "unacquired" correctly by the user (adding info in the acquire/unacquire methods like __FILE__ and __LINE__ could help narrow the bug search).

Thus the manager will be able to:

1. NOT free a non-existing object (how did the C user managed to acquire one, by the way ?)
2. KNOW at the end of execution which objects were not unaquired
3. In case of unacquired objets, destroy them anyway (which is good from a RAII viewpoint) This is somewhat evil, but you could offer this
4. As shown in the code above, it could even help detect a pointer does not point to a valid class