How to read from large process output correctly

问题

There are already a few answers on how to read from a process stream but as far as I can see they do not cover reading from a process which:

may run for a long time before making any output
makes a huge output at the end of its lifetime (more than the stream buffer can hold)
should not run longer than a given time frame
the resulted output is needed as a whole (long character string)

Therefore using the mentioned solutions would either result in wasted process cycles as the loop will try to read the stream even though there is no output, or the non termination of the solution as the process cannot print its whole output (due to full buffer) and the output handler waits for process termination prior to reading.

My current solution looks like this (inspired by an solution for bulk reading large files)

(defun control-process (process timeout)
  (sb-ext:with-timeout timeout
    (handler-case
      (do ((output-stream (sb-ext:process-output process))
           (string nil))
          ((and (equalp (sb-ext:process-status process) :exited)
            (equalp (peek-char nil output-stream nil :eof) :eof))
           (values string :exited))
        (cond 
          ((equalp (sb-ext:process-status process) :signaled)
           (error 'unexpected-process-finish :finish-status :signaled))
          ((equalp (sb-ext:process-status process) :stopped)
           (error 'unexpected-process-finish :finish-status :stopped)))
        (let ((seq (make-string (file-length output-stream))))
          (read-sequence seq output-stream)
          (setf string (concatenate 'string string seq))
          (sleep 1)))
     (sb-ext:timeout (err)
      (declare (ignore err))
      (values nil :timeout))
     (unexpected-process-finish (err)
      (values nil (finish-status err))))))

The function is called with a process:

(sb-ext:run-program "/path/to/programm"
                           (list "--params" "foo" "bar") 
                           :output :stream :wait nil)

But this solution has its drawbacks:

it does not work as the stream is not associated with a file (error)
it does a generic sleep of 1 even though there might not be a output at that time
it does a lot of concatenation which seems to an inelegant solution

Final handling/clean-up of an exited/stopped/too long running process is handled by the calling function.

How can I read from a process which:

may run for a long time (and does its output at the end of its lifetime)
may make a output larger than the stream buffer
must not run longer than a given time span
output is needed as a whole

回答1:

Might a string output stream work for you? The output will be stored in the string that you get back afterward, so the buffering shouldn't be too much of a problem. E.g.,

* (with-output-to-string (out)
    (sb-ext:run-program "/bin/ls" '("/") :output out))

"bin
boot
…
vmlinuz
vmlinuz.old
"

If you want to preallocate the string, you can do that to, with with-output-to-string's string-form argument.

You don't have to use with-output-to-string to use a string-output-stream, though. You could also create one with make-string-output-stream and pass it to sb-ext:run-program. You'd get the text out of it eventually with get-output-stream-string.

来源：https://stackoverflow.com/questions/24308919/how-to-read-from-large-process-output-correctly

标签

common-lisp

sbcl