Don't use struct for "complex" return types, prefer something like this:

C++:

extern "C" __declspec(dllexport) void __stdcall TestMethod2(RGB *prgb) {
    prgb->R = 1;
    prgb->G = 2;
    prgb->B = 3;
}

C#:

[DllImport(@"D:\smo\source\repos\ConsoleApplication4\Debug\Dll1.dll")]
static extern void TestMethod2(ref RGB rgb);

static void Main(string[] args)
{
    var rgb = new RGB();
    TestMethod2(ref rgb);
}

Note in your particular case, it fails because the structure size is 3, so you can make it work if you change the structure like this for example:

C++:

typedef struct _RGB {
    unsigned char R, G, B, A;
} RGB;

C#

[StructLayout(LayoutKind.Sequential)]
struct RGB { public byte R, G, B, A; }

With this definition, the size will be 4, so the C++ compiler will generate a code that will return an int32 value instead of returning - probably - a reference to some internal memory that will be gone by the time execution reaches the .NET side. This is pure luck (or hack) and depends on the C++ compiler I guess.