I have an application that calls some other utility application to set some settings for a particular device. That utility application is called using ShellExecuteEx.
So
You have two things to simulate: ownership and modality.
To simulate ownership: You need to set the owner of your new child process window to your window. This should alleviate any z ordering issues. Though I don't know if this works from another process. If not then you might have to attach your thread input queues and then call it. Or use some other code injection technique.
SetWindowLong , GWL_HWNDPARENT,
To simulate modality, I think you are on the right track with EnableWindow and the WaitForSingleObjectEx.