Since .NET sees the COM object is a .NET object, it cannot make an equivalence between the two definitions of the IComClass1 interface, hence the invalid cast exception. One solution is to declare the common COM interfaces in a separate .tlb file (type library) from an .idl file.

To create a .TLB you could imagine to export the .NET component as a TLB using regasm, unfortunately it works, but the resulting .TLB cannot be referenced by a .NET project (again, no .NET->COM->.NET).

However it's still a good start as you can have a look at the resulting .TLB with OleView from the Windows SDK and basically remove all "custom" IDL attributes that tell it's a .NET exported .tlb.

So, let's say I have this definition in C#:

namespace ClassLibrary2
{
    [InterfaceType(ComInterfaceType.InterfaceIsDual)]
    [Guid("EAA4976A-45C3-4BC5-BC0B-E474F4C3C83F")]
    [ComVisible(true)]
    public interface IComClass1
    {
        void Do();
    }
}

I compile and run this for example (or use TlbExp):

c:\MyProject\ClassLibrary2\bin\Debug>tlbexp ClassLibrary2.dll
Microsoft (R) .NET Framework Assembly to Type Library Converter 4.8.3928.0
Copyright (C) Microsoft Corporation.  All rights reserved.

Assembly exported to 'c:\MyProject\ClassLibrary2\bin\Debug\ClassLibrary2.tlb'

Here is what it look like in OleView:

If you try to reference this from a .NET project it will say:

---------------------------
Microsoft Visual Studio
---------------------------
A reference to 'c:\MyProject\ClassLibrary2\bin\Debug\ClassLibrary2.tlb' could not be added. Please make sure that the file is accessible, and that it is a valid assembly or COM component.
---------------------------
OK   
---------------------------

This is really dumb IMHO. Anyway, so, you have to modify the .tlb. Copy paste the OleView result in some test.idl text file, remove all those "custom" idl attributes, something like this:

[
  uuid("your tlb guid here"),
  version(1.0),

]
library ClassLibrary2 // some name or keep ClassLibrary2
{
    importlib("mscorlib.tlb");
    importlib("stdole2.tlb");

    [
      odl,
      uuid(EAA4976A-45C3-4BC5-BC0B-E474F4C3C83F), // your interface id
      version(1.0),
      dual,
      oleautomation,

    ]
    interface IComClass1 : IDispatch { // if the interface is IDispatch
        [id(0x60020000)]
        HRESULT Do();
    };
};

Now run the MIDL compiler on the test.idl file:

c:\MyProject\ClassLibrary2\ClassLibrary1>midl test.idl
Microsoft (R) 32b/64b MIDL Compiler Version 8.01.0622
Copyright (c) Microsoft Corporation. All rights reserved.
Processing .\test.idl
test.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\oaidl.idl
oaidl.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\objidl.idl
objidl.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\unknwn.idl
unknwn.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\wtypes.idl
wtypes.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\wtypesbase.idl
wtypesbase.idl
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\basetsd.h
basetsd.h
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\guiddef.h
guiddef.h
Processing C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\oaidl.acf
oaidl.acf

This will create a .TLB then you can now reference in both your projects using standard .NET tooling. Note the types it contains should be automatically embedded in your assemblies, no need to ship the .tlb or any other binary.