First of all, if you want to prevent application crash with the code above, you'll need to use SetUnhandledExceptionFilter, like this:

LONG WINAPI MyUnhandledExceptionFilter(struct _EXCEPTION_POINTERS *exceptionInfo)
{
    // do something useful
    return EXCEPTION_EXECUTE_HANDLER; // prevent crash
}

int _tmain(int argc, _TCHAR* argv[])
{
    SetUnhandledExceptionFilter(MyUnhandledExceptionFilter);
        ...
}

But this may not be what you really want. One solution (as proposed by Polity I believe) is to create an intermediary AppDomain that can catch easily all unhandled exceptions. You can do that in C#, like this:

public class PluginVerifier
{
    public static int CheckPlugin(string arguments)
    {
        AppDomain appDomain = AppDomain.CreateDomain(Guid.NewGuid().ToString());
        appDomain.UnhandledException += AppDomainUnhandledException;
        object obj = appDomain.CreateInstanceAndUnwrap("ExceptionThrower", "ExceptionThrower.MainExceptionThrower");
        object ret = obj.GetType().InvokeMember("Startup", BindingFlags.Instance | BindingFlags.Public | BindingFlags.InvokeMethod, null, obj, new object[] { arguments });
        AppDomain.Unload(appDomain);
        return (int)ret;
    }

    private static void AppDomainUnhandledException(object sender, UnhandledExceptionEventArgs e)
    {
        AppDomain appDomain = (AppDomain)sender;
        // the following will prevent "this should never print" to happen
        AppDomain.Unload(appDomain);
    }
}

For this to be able to work however, you need to do two changes to your plugin classes:

• they must derive from MarshalByRefObject
• the plugin method must not be static (static methods call do not go through AppDomain filter)

So your class would be written like this:

public class MainExceptionThrower: MarshalByRefObject
{
    public int StartUp(string arguments)
    {
    ...
    }
 }

If you do this, you can remove the calls to SetUnhandledExceptionPolicy, SetActionOnFailure, or SetDefaultAction, and just replace the bootstrap code like this:

    hr = runtimeHost->ExecuteInDefaultAppDomain(L"PluginSystem.dll", L"PluginSystem.PluginVerifier", L"CheckPlugin", L"test", &returnVal);        

If you try this with your Startup code above, this call will return hr=0x80131604, which is COR_E_TARGETINVOCATION (TargetInvocationException).