How to ensure that we read all lines from boost::child process

Question

I saw the following code on boost::child documentation page where they explain how to read the output of a child process. http://www.boost.org/doc/libs/1_64_0/doc/h

Answer 1

I had the same issue... The best way to deal with this is by using async i/o.

Unfortunately, the boost documentation @ http://www.boost.org/doc/libs/master/doc/html/boost_process/extend.html#boost_process.extend.async was wrong... It makes it all look simple, but doesn't show that the buffers must be sized beforehand, and glosses over many details.

Here's my function, which send or receives one buffer in one go (no interaction like question/response0, I use stderr for error checking, since that's what was needed for my app, but you could catch the app's exit code by calling 'c.exit_code();`.

using tstring=basic_string<TCHAR>;

void Run(
    const tstring& exeName;
    const tstring& args,
    const std::string& input,
    std::string& output,
    std::string& error
)
{
    using namespace boost;

    asio::io_service ios;

    std::vector<char> vOut(128 << 10);
    auto outBuffer{ asio::buffer(vOut) };
    process::async_pipe pipeOut(ios);

    std::function<void(const system::error_code & ec, std::size_t n)> onStdOut;
    onStdOut = [&](const system::error_code & ec, size_t n)
    {
        output.reserve(output.size() + n);
        output.insert(output.end(), vOut.begin(), vOut.begin() + n);
        if (!ec)
        {
            asio::async_read(pipeOut, outBuffer, onStdOut);
        }
    };

    std::vector<char> vErr(128 << 10);
    auto errBuffer{ asio::buffer(vErr) };
    process::async_pipe pipeErr(ios);
    std::function<void(const system::error_code & ec, std::size_t n)> onStdErr;
    onStdErr = [&](const system::error_code & ec, size_t n)
    {
        error.reserve(error.size() + n);
        error.insert(error.end(), vErr.begin(), vErr.begin() + n);
        if (!ec)
        {
            asio::async_read(pipeErr, errBuffer, onStdErr);
        }
    };

    auto inBuffer{ asio::buffer(input) };
    process::async_pipe pipeIn(ios);

    process::child c(
        exeName + _T(" ") + args, 
        process::std_out > pipeOut, 
        process::std_err > pipeErr, 
        process::std_in < pipeIn
    );


    asio::async_write(pipeIn, inBuffer, 
        [&](const system::error_code & ec, std::size_t n) 
        {
            pipeIn.async_close();
        });

    asio::async_read(pipeOut, outBuffer, onStdOut);
    asio::async_read(pipeErr, errBuffer, onStdErr);

    ios.run();
    c.wait();
}

The app I execute is a decoder/encoder, I send an entire file to process, and receive the results simultaneously this way. No file size limit.

IMPORTANT

There is a bug fix in boost 1.64, only affects Windows, apparently

in file boost\process\detail\windows\async_pipe.hpp: ref: https://github.com/klemens-morgenstern/boost-process/issues/90

line 79:

    ~async_pipe()
    {
//fix
        //if (_sink .native()  != ::boost::detail::winapi::INVALID_HANDLE_VALUE_)
        //    ::boost::detail::winapi::CloseHandle(_sink.native());
        //if (_source.native() != ::boost::detail::winapi::INVALID_HANDLE_VALUE_)
        //    ::boost::detail::winapi::CloseHandle(_source.native());
        boost::system::error_code ec;
        close(ec);
//fix
    }

Answer 2

Another solution.

bp::child c(bp::search_path("nm"), file, bp::std_out > output, bp::std_err > error, svc);

This code ignores the order of stdout/stderr. If the order concerns, you can write like this:

bp::child c(bp::search_path("nm"), file, (bp::std_out & bp::std_err) > outerr, svc);

Answer 3

I think there's no proper way unless you use asynchronous methods.

Perhaps you can simple get a future to a vector and use string_views into that if you somehow really need it line-by-line.

std::future<std::vector<char> > output, error;

boost::asio::io_service svc;
bp::child c(bp::search_path("nm"), file, bp::std_out > output, bp::std_err > error, svc);
svc.run();

To read exactly like you did before you can use an istream on top of the vector:

#include <boost/process.hpp>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream_buffer.hpp>
#include <iostream>

namespace bp = boost::process;
namespace bio = boost::iostreams;
std::string const file = "./a.out";

int main() {
    std::future<std::vector<char> > output, error;

    boost::asio::io_service svc;
    bp::child c(bp::search_path("nm"), file, bp::std_out > output, bp::std_err > error, svc);
    svc.run();

    //then later
    {
        auto raw = output.get();
        std::vector<std::string> data;
        std::string line;
        bio::stream_buffer<bio::array_source> sb(raw.data(), raw.size());
        std::istream is(&sb);

        while (std::getline(is, line) && !line.empty())
            data.push_back(line);

        std::cout << data.at(rand()%data.size()) << "\n";
    }

}