I have an application that calls some other utility application to set some settings for a particular device. That utility application is called using ShellExecuteEx.
So
Let's take a look at your approach #3, which is very close to what you want. I suspect the problem is that when the secondary app closes, Windows decides that it doesn't want to restore focus to a disabled window. You could try to re-enable your window before that happens but that's likely to be tricky (and not worth the effort).
Instead of disabling the window directly, try disabling it by just ignoring user input. So rather than calling EnableWindow, change your message loop to filter out input messages. In particular, if
msg >= WM_KEYFIRST || msg <= WM_KEYLAST || msg >= WM_MOUSEFIRST || msg <= WM_MOUSELAST
then discard the message; otherwise, pass it on to the normal dispatch loop. What you're doing is creating your own disabled window, but Windows doesn't know that.