In this case, you don't explicitly need to use P/Invoke since you don't have to pass the struct back and forth between managed and native code. So you could do this instead. It would avoid this useless GC handle allocation, and allocate only what's needed.

public struct PreIndexStruct {
    public string Key;
    public long Offset;
    public int Count;
}

while (...) {
    ...
    PreIndexStruct pis = new PreIndexStruct();
    pis.Key = Encoding.Default.GetString(reader.ReadBytes(96));
    pis.Offset = reader.ReadInt64();
    pis.Count = reader.ReadInt32();
    structures.Add(pis);
}

I'm not sure you can be much faster than this.