See these two articles about it:

AnyCPU Exes are usually more trouble than they're worth

Visual Studio: Why is there no 64 bit version? (yet)

See also this post about 64-bit code: Should I choose to take advantage of 64-bit?, it clearly states that for example Paint.NET works better under in 64-bit mode, because the developer optimized the code for 64-bit.

So, I don't think you are doing something wrong, but maybe the 3rd party .NET assemblies you're using could be improved specifically when running in a 64-bit process.