When you're working with p/invoke, you have to stop thinking "managed", and instead think physical binary layout, 32 vs 64 bit, etc. Also, when the called native binary always runs in-process (like here, but with COM servers it can be different) it's easier than out-of-process because you don't have to think too much about marshaling/serialization, ref vs out, etc.

Also, you don't need to tell .NET what it already knows. An array of float is an LPArray of R4, you don't have to specify it. The simpler the better.

So, first of all flann_index_t. It's defined in C as void *, so it must clearly be an IntPtr (an opaque pointer on "something").

Then, structures. Structures passed as a simple pointer in C can just be passed as a ref argument in C# if you define it as struct. If you define it as a class, don't use ref. In general I prefer using struct for C structures.

You'll have to make sure the structure is well defined. In general, you use the LayoutKind.Sequential because .NET p/invoke will pack arguments the same way that the C compiler does. So you don't have to use explicit, especially when arguments are standard (not bit things) like int, float, So you can remove all FieldOffset and use LayoutKind.Sequential if all members are properly declared... but this is not the case.

For types, like I said, you really have to think binary and ask yourself for each type you use, what's its binary layout, size? int are (with 99.9% C compilers) 32-bit. float and double are IEEE standards, so there should never be issues about them. Enums are in general based on int, but this may vary (in C and in .NET, to be able to match C). long are (with 99.0% C compilers) 32-bit, not 64-bit. So the .NET equivalent is Int32 (int), not Int64 (long).

So you should correct your FlannParameters structure and replace the long by an int. If you really want to make sure for a given struct, check Marshal.SizeOf(mystruct) against C's sizeof(mystruct) with the same C compiler than the one that was used to compile the library you're calling. They should be the same. If they're not, there's an error in the .NET definition (packing, member size, order, etc.).

Here are modified definitions and calling code that seem to work.

static void Main(string[] args)
{
    int rows = 3, cols = 5;
    int tCount = 2, nn = 3;

    float[,] dataset2D = { { 1.0f,      1.0f,       1.0f,       2.0f,       3.0f},
                           { 10.0f,     10.0f,      10.0f,      3.0f,       2.0f},
                           { 100.0f,    100.0f,     2.0f,       30.0f,      1.0f} };

    float[,] testset2D = { { 1.0f,      1.0f,       1.0f,       1.0f,       1.0f},
                           { 90.0f,     90.0f,      10.0f,      10.0f,      1.0f} };

    var fparams = new FlannParameters();
    var index = NativeMethods.flannBuildIndex(dataset2D, rows, cols, out float speedup, ref fparams);

    var indices = new int[tCount, nn];
    var idists = new float[tCount, nn];
    NativeMethods.flannFindNearestNeighborsIndex(index, testset2D, tCount, indices, idists, nn, ref fparams);
    NativeMethods.flannFreeIndex(index, ref fparams);
}

[DllImport(DllWin32, EntryPoint = "flann_build_index", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr flannBuildIndex(float[,] dataset,
                                            int rows, int cols,
                                            out float speedup, // out because, it's and output parameter, but ref is not a problem
                                            ref FlannParameters flannParams);

[DllImport(DllWin32, EntryPoint = "flann_free_index", CallingConvention = CallingConvention.Cdecl)]
public static extern int flannFreeIndex(IntPtr indexPtr,  ref FlannParameters flannParams);

[DllImport(DllWin32, EntryPoint = "flann_find_nearest_neighbors_index", CallingConvention = CallingConvention.Cdecl, SetLastError = true)]
public static extern int flannFindNearestNeighborsIndex(IntPtr indexPtr,
                                                        float[,] testset,
                                                        int tCount,
                                                        [In, Out] int[,] result, // out because it may be changed by C side
                                                        [In, Out] float[,] dists,// out because it may be changed by C side
                                                        int nn,
                                                        ref FlannParameters flannParams);

[StructLayout(LayoutKind.Sequential)]
public struct FlannParameters
{
    public FlannAlgorithmEnum algorithm;
    public int checks;
    public float eps;
    public int sorted;
    public int maxNeighbors;
    public int cores;
    public int trees;
    public int leafMaxSize;
    public int branching;
    public int iterations;
    public FlannCentersInitEnum centersInit;
    public float cbIndex;
    public float targetPrecision;
    public float buildWeight;
    public float memoryWeight;
    public float sampleFraction;
    public int tableNumber;
    public int keySize;
    public int multiProbeLevel;
    public FlannLogLevelEnum logLevel;
    public int randomSeed;
}

note: I've tried to use the specific flann parameters values, but the library crashes in this case, I don't know why...