SetStdHandle has no effect on cout/printf

Question

The title says it all. When I run the following code:

HANDLE hOut = GetStdHandle(STD_OUTPUT_HANDLE);
HANDLE hFile = CreateFile(TEXT(\"Foo.txt\"), GENERIC_WRI         


        

Answer 1

Wow, after a while of searching for a way to manually set the HANDLE of a FILE, I finally discovered that there is a fairly straightforward way to do this using the C Run-Time Library:

std::vector<int> vfdStdOut;
void StartStdOutRedirection(_In_ LPCSTR lpFile)
{
    // Duplicate stdout and give it a new file descriptor
    int fdDup = _dup(_fileno(stdout));
    vfdStdOut.push_back(fdDup);

    // Re-open stdout to the new file
    freopen(lpFile, "w", stdout);
}

bool EndStdOutRedirection(void)
{
    if (vfdStdOut.size() != 0)
    {
        // Get last saved file descriptor and restore it
        int fdNext = vfdStdOut.back();
        _dup2(fdNext, _fileno(stdout));

        // Need to close the file associated with the saved file descriptor
        _close(fdNext);

        vfdStdOut.pop_back();
        return true;
    }

    return false;
}

This will also take care of calling SetStdHandle for you!

Answer 2

Here's a solution I put together (far from perfect of course). It calls a custom function for every character written to STDOUT. In my example, it forwards the stream to OutputDebugString calls.

#include <windows.h>
#include <io.h>
#include <functional>
#include <iostream>

#define STDOUT_FILENO 1
#define STDERR_FILENO 2

enum StdHandleToRedirect {
    STDOUT, STDERR
};

class StdRedirect {
public:
    /// Assumes the specified handle is still assigned to the default FILENO (STDOUT_FILENO/STDERR_FILENO)
    /// TODO allow redirection in every case
    /// callback will run in a new thread and will be notified of any character input to
    /// the specified std handle
    StdRedirect(StdHandleToRedirect h, std::function<void(char)> callback) : callback(callback) {
        CreatePipe(&readablePipeEnd, &writablePipeEnd, 0, 0);
        SetStdHandle(h == STDOUT ? STD_OUTPUT_HANDLE : STD_ERROR_HANDLE, writablePipeEnd);

        // Redirect (TODO: ERROR CHECKING)
        int writablePipeEndFileStream = _open_osfhandle((long)writablePipeEnd, 0);
        FILE* writablePipeEndFile = NULL;
        writablePipeEndFile = _fdopen(writablePipeEndFileStream, "wt");
        _dup2(_fileno(writablePipeEndFile), h == STDOUT ? STDOUT_FILENO : STDERR_FILENO);

        CreateThread(0, 0, (LPTHREAD_START_ROUTINE)stdreader, this, 0, 0);
    }

    // TODO implement destructor, cleanup, reset

private:
    // DWORD (WINAPI *PTHREAD_START_ROUTINE)(LPVOID lpThreadParameter)
    static void WINAPI stdreader(StdRedirect* redirector) {
        while (1) {
            char c;
            DWORD read;
            ::fflush(NULL); // force current stdout to become readable
            // TODO add error handling
            ReadFile(redirector->readablePipeEnd, (void*)&c, 1, &read, 0); // this blocks until input is available
            if (read == 1)
                redirector->callback(c);
        }
    }

    HANDLE readablePipeEnd, writablePipeEnd;
    const std::function<void(char)> callback;
};

int main() {
    std::function<void(char)> toOutputDebugString = [](char x) {
        char str[2] = {x, 0};
        OutputDebugStringA(str);
    };

    StdRedirect so(STDOUT, toOutputDebugString);
    std::cout << "test stdout\n";
    while (1); // busy loop to give the thread time to read stdout. 
    // You might want to look at "Output: Show output from: Debug" now.
    return 0;
}

Answer 3

This would work only for MS-CRT.

FILE* are just entries in a an array of FILE struct inside the thread local storage.

So my idea would be:

1. Open the new file with fopen. We now have a new FILE* to the internal structure array.
2. Save this new pointer into your stack.
3. Now just swap the two structures stdout with the new FILE struct.

Code should be:

FILE swap = *stdout;
*stdout = *pFile;
*pFile = swap;

After this operation the stdout handle is now the new file. The old standard out handle is in the FILE* slow you saved on the stack.

To return just:

1. Get the FILE+ from the stack.
2. swap stdout again with this FILE*. (swap complete FILE structs)
3. Close the FILE* you received.

If you want to do this with file handles, you need to associate the OS file handle to a FILE*. This is done with _open_osfhandle() and _fdopen().

Because file operations use a buffer, you need to flush the buffers before the swap. To make sure that there are no "leftovers" from an old output.

Form my point of view this hack should work.