How to use istream with strings

Question

I would like to read an file into a string. I am looking for different ways for how to do it efficiently.

Using a fixed size *char buffer

I

Answer 1

The most general way would be probably be the response using the istreambuf_iterator:

std::string s( (std::istreambuf_iterator<char>( source )),
               (std::istreambuf_iterator<char>()) );

Although exact performance is very dependent on the implementation, it's highly unlikely that this is the fastest solution.

An interesting alternative would be:

std::istringstream tmp;
tmp << source.rdbuf();
std::string s( tmp.str() );

This could be very rapid, if the implementation has do a good job on the operator<< you're using, and in how it grows the string within the istringstream. Some earlier implementations (and maybe sone more recent ones as well) were very bad at this, however.

In general, performance using an std::string will depend on how efficient the implementation is in growing a string; the implementation cannot determine how large to make it initially. You might want to compare the first algorithm using the same code with std::vector<char> instead of std::string, or if you can make a good estimate of the maximum size, using reserve, or something like:

std::string s( expectedSize, '\0' );
std::copy( std::istreambuf_iterator<char>( source ),
           std::istreambuf_iterator<char>(),
           s.begin() );

memcpy cannot read from a file, and with a good compiler, will not be as fast as using std::copy (with the same data types).

I tend to use the second solution, above, with the << on the rdbuf(), but that's partially for historical reasons; I got used to doing this (using istrstream) before the STL was added to the standard library. For that matter, you might want to experiment with istrstream and a pre-allocated buffer (supposing you can find an appropriate size for the buffer).

Answer 2

it only happens in the memory, not from the disk, so it is almost unnoticable

That is indeed correct. Still, a solution that doesn’t do that may be faster.

Why are those iterators so slow?

The code is slow not because of the iterators but because the string doesn’t know how much memory to allocate: the istreambuf_iterators can only be traversed once so the string is essentially forced to perform repeated concatenations with resulting memory reallocations, which are very slow.

My favourite one-liner, from another answer is streaming directly from the underlying buffer:

string str(static_cast<stringstream const&>(stringstream() << in.rdbuf()).str());

On recent platforms this will indeed pre-allocate the buffer. It will however still result in a redundant copy (from the stringstream to the final string).