I've only tested what follows on .NET 8 and above.

Considering you have a proper regfree setup (which is far from being easy...), what happens is for some reason in regfree COM, the generated COMServer.comhost.dll doesn't answer to direct calls for IDispatch interfaces.

Notable clients that create a COM coclass like this are: VB6/VBA (so Office/Excel macros), VBScript.

So for example, from C++, if the client does this:

IDispatch* disp;
auto hr = CoCreateInstance(
    __uuidof(COMServer), 
    nullptr, 
    CLSCTX_ALL, 
    __uuidof(IDispatch),
    (void**)&disp); 

hr value will be E_NOINTERFACE (which is the same as "Specified cast is not valid" in .NET parlance).

While first asking for IUnknown and then calling QueryInterface for IDispatch works fine.

The trick is for example to declare the interface like this (note the Dual implementation which .NET 8+ does support):

[ComImport, Guid("38268d0f-eae8-4c95-bac7-9d96c083ea7c"), InterfaceType(ComInterfaceType.InterfaceIsDual)]
public interface IServer
{
    double ComputePi();
}

And the class like this :

[ComVisible(true), Guid("cd42c360-c635-44fe-a8ab-793d188512a9"), ProgId("RegfreeNetCom.Server"), ClassInterface(ClassInterfaceType.None)]
public class Server : IServer, Server.IDispatch
{
    public double ComputePi()
    {
        return Math.PI;
    }

    // This is not needed with normal COM registration, and this is not needed in out-of-process context
    // For some reason, regfree COM and .NET Core does not work without an explicit IDispatch in a coclass such as this one
    // What's weird is it doesn't work only in the case where IDispatch is directly requested in CoCreateInstance call (it works if IUnkown is queried first and IDispatch is requested later).
    // Querying IDispatch first is used (at least) by VB/VBA/VBScript clients.
    // Note however, you *must* declare IServer as dual interface, otherwise it will proably crash since this IDispatch is not defined at all.
    [ComImport, Guid("00020400-0000-0000-c000-000000000046"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    private interface IDispatch
    {
    }
}

This "private" IDispatch should never be called, but it's needed.

There's a full sample code here: https://github.com/smourier/RefreeNetCom including VB6, Excel VBA and VBScript clients examples.