Here is solution that does not require any coding and works even for texts of exceptions that are loaded too early for us to be able to change by code (for example, those in mscorlib).

It may not be always applicable in every case (it depends on your setup as you need to be able to create a .config file aside the main .exe file) but that works for me. So, just create an app.config in dev, (or a [myapp].exe.config or web.config in production) that contains the following lines for example:

<configuration>
  ...
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="mscorlib.resources" publicKeyToken="b77a5c561934e089"
                          culture="fr" /> <!-- change this to your language -->

        <bindingRedirect oldVersion="1.0.0.0-999.0.0.0" newVersion="999.0.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Xml.resources" publicKeyToken="b77a5c561934e089"
                          culture="fr" /> <!-- change this to your language -->

        <bindingRedirect oldVersion="1.0.0.0-999.0.0.0" newVersion="999.0.0.0"/>
      </dependentAssembly>

      <!-- add other assemblies and other languages here -->

    </assemblyBinding>
  </runtime>
  ...
</configuration>

What this does is tell the framework to redirect assembly bindings for mscorlib's resources and System.Xml's resources, for versions between 1 and 999, in french (culture is set to "fr") to an assembly that ... does not exists (an arbitrary version 999).

So when the CLR will look for french resources for these two assemblies (mscorlib and System.xml), it will not find them and fallback to English gracefully. Depending on your context and testings, you might want to add other assemblies to these redirects (assemblies that contains localized resources).

Of course I don't think this is supported by Microsoft, so use at your own risk. Well, in case you detect a problem, you can just remove this configuration and check it's unrelated.