MSI installation interrupted by custom action DLL

Question

I have used a Visual Studio setup project to create an MSI. I have edited the MSI in Orca so that a custom action is performed via a DLL when it is first opened. When I run the MSI, msiexec logs the following:

MSI (c) (E4:BC) [15:28:14:453]: Doing action: CustomAction1
Action 15:28:14: CustomAction1. 
Action start 15:28:14: CustomAction1.
MSI (c) (E4:BC) [15:28:14:453]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'CustomAction1' 
MSI (c) (E4:BC) [15:28:14:453]: Creating MSIHANDLE (13) of type 790542 for thread 3260
MSI (c) (E4:B4) [15:28:14:453]: Invoking remote custom action. DLL: C:\DOCUME~1\USERNA~1\LOCALS~1\Temp\MSIA3.tmp, Entrypoint: SampleFunction
MSI (c) (E4:B4) [15:28:14:453]: Closing MSIHANDLE (13) of type 790542 for thread 3260
Action ended 15:28:14: CustomAction1. Return value 3.
MSI (c) (E4:BC) [15:28:14:468]: Doing action: FatalErrorForm
Action 15:28:14: FatalErrorForm. 
Action start 15:28:14: FatalErrorForm.
MSI (c) (E4:BC) [15:28:14:468]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'FatalErrorForm' 

The installer wizard then displays the error message: The installer was interrupted before MyProduct could be installed. You need to restart the installer to try again. The installer was interrupted before MyProduct could be installed. You need to restart the installer to try again.

The custom DLL is written in C++. Here is the source code:

MyCustomAction.cpp:

// MyCustomAction.cpp : Defines the entry point for the DLL application.
#include "stdafx.h"

BOOL APIENTRY DllMain(HANDLE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved)
{
    return TRUE;
}

UINT __stdcall SampleFunction(MSIHANDLE hModule)
{
        //This is the function that is called by the MSI
        //It is empty because I just want to check that it can be called without interrupting the installer, then I will add the actual functionality
}

MyCustomAction.def:

; MyCustomAction.def
;
; defines the exported functions which will be available to the MSI engine

LIBRARY      "MyCustomAction" 
DESCRIPTION  'Custom Action DLL'

EXPORTS
    SampleFunction

I have also referenced msi.lib in the DLL's additional dependencies. Why does the custom action interrupt the installation when I am currently not explicitly telling it do anything? Any help would be appreciated.

UPDATE:

In Orca, the custom action is in the Binary table and is type 1 in the CustomAction table. The custom action is Immediate and takes place after IsolateComponents and before WelcomeForm in the InstallUISequence table.

Answer 1

You should update your code and your post to indicate that you are actually returning ERROR_SUCCESS. Whether it fixes the issue or not, the point is that it is the correct thing to do. If it doesn't return a value the call sequence will fail and get an error.

Your Dll is probably not loading because of missing dependencies. If you put a simple messagebox call in your code you'll at least see if the code is actually starting to run. C++ will require runtime support Dlls that may not be on the system already.

If it turns out that you have dependencies on C++ runtimes then you'll need to install them before running your MSI. That's what the prerequisites choice is for - it generates a setup.exe to install the dependencies and then installs your MSI.