Technique for using std::ifstream, std::ofstream in python via SWIG?

Question

Is there a way to use std::[io]fstream\'s in python via swig?

I have a c-class with functions like:

void readFrom(std::istream& istr         


        

Answer 1

My preferred solution to this problem would be to make the interface exposed to Python developers as "Pythonic" as possible. In this instance that would be to accept python file objects as your ostream and istream arguments.

To achieve that we have to write a typemap to set up each mapping.

I've written the following header file to demonstrate this in action:

#ifndef TEST_HH
#define TEST_HH
#include <iosfwd>

void readFrom(std::istream& istr);
void writeTo(std::ostream& ostr);
#endif

Which I wrote a dummy implementation for testing as:

#include <iostream>
#include <cassert>
#include "test.hh"

void readFrom(std::istream& istr) {
  assert(istr.good());
  std::cout << istr.rdbuf() << "\n";
}

void writeTo(std::ostream& ostr) {
  assert(ostr.good());
  ostr << "Hello" << std::endl;
  assert(ostr.good());
}

With that in place I was able to wrap it successfully using:

%module test

%{
#include <stdio.h>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
namespace io = boost::iostreams;
typedef io::stream_buffer<io::file_descriptor_sink> boost_ofdstream;
typedef io::stream_buffer<io::file_descriptor_source> boost_ifdstream;
%}

%typemap(in) std::ostream& (boost_ofdstream *stream=NULL) {
  int fd = -1;

  #if PY_VERSION_HEX >= 0x03000000
  fd = PyObject_AsFileDescriptor($input);
  #else 
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (f) fd = fileno(f);
  #endif
  if (fd < 0) {
    SWIG_Error(SWIG_TypeError, "File object expected.");
    SWIG_fail;
  }
  else {
    // If threaded incrment the use count
    stream = new boost_ofdstream(fd, io::never_close_handle);
    $1 = new std::ostream(stream);
  }
}

%typemap(in) std::istream& (boost_ifdstream *stream=NULL) {
  int fd = -1;

  #if PY_VERSION_HEX >= 0x03000000
  fd = PyObject_AsFileDescriptor($input);
  #else 
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (f) fd = fileno(f);
  #endif
  if (fd < 0) {
    SWIG_Error(SWIG_TypeError, "File object expected.");  
    SWIG_fail;
  }
  else {
    stream = new boost_ifdstream(fd, io::never_close_handle);
    $1 = new std::istream(stream);
  }
}

%typemap(freearg) std::ostream& {
  delete $1;
  delete stream$argnum;
}

%typemap(freearg) std::istream& {
  delete $1;
  delete stream$argnum;
}

%{
#include "test.hh"
%}
%include "test.hh"

The core bit of this is basically calling PyFile_AsFile() to get a FILE* from the Python file object. With that we can then construct a boost object that uses a file descriptor as the source/sink as appropriate.

The only thing that remains is to clean up the objects we created after the call has happened (or if an error prevented the call from happening).

With that in place we can then use it as expected from within Python:

import test
outf=open("out.txt", "w")
inf=open("in.txt", "r")

outf.write("Python\n");

test.writeTo(outf)
test.readFrom(inf)

outf.close()
inf.close()

Note the buffering semantics might not produce the results you expected, for instance in out.txt I get:

Hello
Python

which is the opposite order of the calls. We can fix that also by forcing a call to file.flush() on the Python file object in our typemap, before constructing a C++ stream:

%typemap(in) std::ostream& (boost_ofdstream *stream=NULL) {
  PyObject_CallMethod($input, "flush", NULL);
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (!f) {
    SWIG_Error(SWIG_TypeError, "File object expected.");
    SWIG_fail;
  }
  else {
    // If threaded incrment the use count
    stream = new boost_ofdstream(fileno(f), io::never_close_handle);
    $1 = new std::ostream(stream);
  }
}

Which has the desired behaviour.

Other notes:

1. If you've got multithread code and the C++ calls are happening without the GIL you'll need to call PyFile_IncUseCount and PyFile_DecUseCount in the in and freearg typemaps respectively to make sure that nothing can close the file whilst you're still using it.
2. I've assumed that PyFile_AsFile returns NULL if the object it's given isn't a file - the documentation doesn't seem to specify that either way, so you could use PyFile_Check to be sure.
3. If you wanted to be super flexible you could accept strings from Python and construct a std::ifstream as appropriate using PyString_Check/PyFile_Check to decide which action to take in the typemap.
4. Some C++ standard libraries provide an ifstream/ofstream constructor which takes FILE*, as an extension. If you have one of those you could use it instead of relying on boost.

Answer 2

I don't know swig but assuming you need to create a copyable object, you might get away using a function like

std::shared_ptr<std::ostream> make_ostream(std::string const& filename) {
    return std::make_shared<std::ofstream>(filename);
}

... and then use a forwarding function to call the function you actually want to call:

void writeTo(std::shared_ptr<std::ostream> stream) {
    if (stream) {
        writeTo(*stream);
    }
}

(if overloading the names causes issues, you could call the forwarding function differently, of course).

Answer 3

I ended up just writing my own proxy class for use within the interface. So I used SWIG to wrap this class:

 /**
 * Simple class to expose std::streams in the python
 * interface.  works around some issues with trying to directy
 * the file stream objects
 */
class ifstream_proxy: boost::noncopyable{
    public:
        ifstream_proxy(): m_istr(){
            // no op
        }
        virtual ~ifstream_proxy(){
            // no op
        }
        void open(const std::string& fname ){
            m_istr.close(); 
            m_istr.open( fname.c_str(), std::ifstream::in|std::ifstream::binary) ;
        }
        std::istream& stream(){
            return m_istr;
        }
        // TBD: do I want to  add additional stream manipulation functions?
   private: 
        std::ifstream m_istr;
};

and in python call make the calls

>>> proxy=ifstream_proxy()
>>> proxy.open('file_to_read_from.txt')
>>> readFrom( stream_proxy.stream() )

Answer 4

I made a few tweaks to the solution provided by @Flexo

The main change was to use a boost::iostream::stream owned by a unique pointer

Some other differences:

• used a fragment for the iostream header requirements
• removed the "io" namespace alias to avoid conflicts with code where that name is already used
• took advantage of fileno() returning -1 for NULL or invalid FILE* arguments
• use "%#include" "%#else" "%#endif" so the directives are in the wrap file and the same wrap file can be used for python2 and python3
• included his suggestion for flushing the output stream because it violated the principle of least surprise without it
• call Py_DECREF on the result from the python output stream flush() method call - this result is None so the world won't catch fire if None's reference counts grow but it's cleaner and guards against overflowing the refcount

%fragment("iostream_header", "header") %{
#include <stdio.h>
#include <memory.h>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
using boost_ofd_stream = boost::iostreams::stream<boost::iostreams::file_descriptor_sink>;
using boost_ifd_stream = boost::iostreams::stream<boost::iostreams::file_descriptor_source>;
%}  

%typemap(in, fragment="iostream_header") std::ostream& (std::unique_ptr<boost_ofd_stream> stream) {
    PyObject *flush_result = PyObject_CallMethod($input, const_cast<char*>("flush"), nullptr);
    if (flush_result) Py_DECREF(flush_result);
%#if PY_VERSION_HEX < 0x03000000
    int fd = fileno(PyFile_AsFile($input));
%#else
    int fd = PyObject_AsFileDescriptor($input);
%#endif
    if (fd < 0) { SWIG_Error(SWIG_TypeError, "File object expected."); SWIG_fail; }
    stream = std::make_unique<boost_ofd_stream>(fd, boost::iostreams::never_close_handle);
    $1 = stream.get();
}   

%typemap(in, fragment="iostream_header") std::istream& (std::unique_ptr<boost_ifd_stream> stream) {
%#if PY_VERSION_HEX < 0x03000000
    int fd = fileno(PyFile_AsFile($input));
%#else
    int fd = PyObject_AsFileDescriptor($input);
%#endif
    if (fd < 0) { SWIG_Error(SWIG_TypeError, "File object expected.");  SWIG_fail; }
    stream = std::make_unique<boost_ifd_stream>(fd, boost::iostreams::never_close_handle);
    $1 = stream.get();
}   

Answer 5

Working .i file based on Dietmar's suggestion to use shared pointers:

%module ptrtest

%include "boost_shared_ptr.i"
%include "std_string.i"

%shared_ptr( std::ostream )

%{
#include <iostream>
#include <fstream>
#include <boost/shared_ptr.hpp>

typedef boost::shared_ptr< std::ostream > ostream_ptr;

ostream_ptr mk_out(const std::string& fname ){
    return ostream_ptr( new std::ofstream( fname.c_str() ) );
}

void writeTo(std::ostream& ostr){
    ostr<<"OK"<<std::endl;
}

%}

namespace std{
    class ostream{
    public:
        // I think we could expose things like write,put here
        // and actually make this class useful from python
    protected:
        // Need to declare this as protected otherwise swig tries 
        // to make/use a public default constructor.
        ostream();
    };
}


// Tell swig to put these into the interface
typedef boost::shared_ptr< std::ostream > ostream_ptr;
ostream_ptr mk_out(const std::string& fname );
void writeTo(std::ostream& ostr);

// Usage:
//>>>ostr=mk_out('/path/to/file.txt')
//>>>writeTo(ostr) # no need to cast/call-function!