Imagine a pure .NET application which do not uses COM components nor PInvoke. Does it matters if the target system is 32 or 64 bits?
Another way it can matter, apart from those identified by Reed Copsey, is if your "pure" application happens to use the System.IntPtr struct for something, or if it uses unsafe code (which isn't the same thing as P/Invoke) and pointer arithmetic.
Another big one to look out for would be calls to pretty much anything in the System.Runtime.InteropServices.Marshal class. There are all kinds of awesome ways to shoot yourself in the foot in there (extremely useful when you need it, of course).