How can I make a child process window to appear modal in my process?

Question

I have an application that calls some other utility application to set some settings for a particular device. That utility application is called using ShellExecuteEx.

So

Answer 1

The short answer is that there is no way to seamlessly make a window in thread B modal for a window in thread A, even if the threads are in the same process. If you own the code for both windows, you may be able to come close, but in that case you will achieve much better results for the effort by putting all of your UI in one thread.

If you try to suggest to the user that thread B's window is modal for thread A's, there are a lot of subtle Z-order and activation behaviors you have to get right (as you have noticed) lest you suffer an uncanny-valley effect of sorts, where it's clear to the user that thread B's window is trying to be something it's not and therefore seems broken.

To avoid that, I would take this approach:

1. The user clicks on "FDA Inspection" in canner.exe's main window. canner.exe shows a modal dialog indicating that it is opening an external program ("Opening Botulism Settings..."). This disables the main window, etc. so that the user knows a modal interaction is taking place.
2. canner.exe calls ShellExecuteEx() to start botulism.exe.
3. canner.exe calls WaitForInputIdle() on the handle returned from ShellExecuteEx(). WaitForInputIdle() will return (approximately, but usually close enough) when botulsim.exe is ready for user interaction. If botulism.exe typically takes five or more seconds to show its UI, I may use a short timeout with WaitforInputIdle() in a loop and occasionally process any pending messages with PeekMessage()/ProcessMessage().
4. canner.exe changes its dialog text to reflect that it is waiting for the user to close botulism.exe ("Close Botulism Settings to continue...").
5. canner.exe calls MsgWaitForMultipleObjects() in a loop to wait until botulsim.exe closes. MsgWaitForMultipleObjects() will return when the handles passed are signaled or when there are messages waiting in the thread's queue.
6. If the user clicks the close box in canner.exe's modal dialog while canner.exe is waiting, canner.exe prompts the user that botulism.exe is still running ("Botulism Settings is still open, continue anyway?", "Yes, I know" or "No, I'm not done"). If confirmed, canner.exe closes the dialog and cancels the original FDA inspection started in step 1 and returns to the main window's message loop.
7. When MsgWaitForMultipleObjects() indicates that botulism.exe is finished, canner.exe closes the dialog and continues normally with the FDA inspection started in step 1.

This way, if everything proceeds normally and quickly, the interaction may well be seamless, but if something goes wrong with the child process or the Z-order gets changed, etc. it will be clear why the parent process is waiting and what the user needs to do to either cancel or go on with the task he started.