MFC application crashing in ProcessShellCommand() when file to open specified on command line

Question

The problem I need to solve is how to use the MFC function ProcessShellCommand() in the InitInstance() of a CWinApp to process a File

Answer 1

I just had the same problems in Windows 10 with Visual Studio 2013 and an MDI app. Here, the provided solution of calling ProcessShellCommand() twice still resulted in a crash. The solution was to create the window earlier than interpreting the command line. That worked for me. I tried the CoInitialize() variant and that also works (put it somewhere before the code below):

// create main MDI Frame window
CMainFrame* pMainFrame = new CMainFrame;
if (!pMainFrame->LoadFrame(IDR_MAINFRAME))
 return FALSE;
m_pMainWnd = pMainFrame;

// The main window has been initialized, so show and update it.
// This needs to be up really before parsing the command line,
// so that any FileOpen command has something to render in.
pMainFrame->ShowWindow(m_nCmdShow);
pMainFrame->UpdateWindow();

// Parse command line for standard shell commands, DDE, file open
CCommandLineInfo cmdInfo;

ParseCommandLine(cmdInfo);
if(!ProcessShellCommand(cmdInfo))
  return FALSE;

Answer 2

After using Visual Studio 2013 to generate a new MFC MDI (Multiple Document Interface) application to compare between the application with which I am having problems launching and the new, generated source code, I have a solution.

The main difference between launching properly and not launching properly seems to be a requirement for initializing COM. The following specific source code was put into the InitInstance() of the application being launched and the application is now working successfully. Part of the source code changes is a call to initialize COM.

// InitCommonControlsEx() is required on Windows XP if an application
// manifest specifies use of ComCtl32.dll version 6 or later to enable
// visual styles.  Otherwise, any window creation will fail.
INITCOMMONCONTROLSEX InitCtrls;
InitCtrls.dwSize = sizeof(InitCtrls);
// Set this to include all the common control classes you want to use
// in your application.
InitCtrls.dwICC = ICC_WIN95_CLASSES;
InitCommonControlsEx(&InitCtrls);

CWinApp::InitInstance();

// Initialize OLE libraries
if (!AfxOleInit())
{
    AfxMessageBox(IDP_OLE_INIT_FAILED);
    return FALSE;
}

AfxEnableControlContainer();

// AfxInitRichEdit2() is required to use RichEdit control   
// AfxInitRichEdit2();

While the Visual Studio 2005 compiled application did not demonstrate this problem, I do want to keep the source as compiled by Visual Studio 2005 and Visual Studio 2013 as similar as possible. I made the same source change in the Visual Studio 2005 source tree and it works properly in the Visual Studio 2005 source tree as well.

Using Visual Studio 2005 and creating an empty MFC application for MDI generates source code similar to the above.

Answer 3

I'm updating a desktop application from VC++ to Visual Studio 2017 and encountered the same problem when the user was trying to open a File with a double click from the Explorer. In my case, I just had to add this code:

// Initialize OLE libraries
if (!AfxOleInit())
{
    AfxMessageBox("Could not open the file! \nTry open CS Setup first and then open the file using the menu \"File->Open...\".", MB_ICONERROR);
    return FALSE;
}