You should compile using the binaries.
The reason is very simple: your project and third-party libraries are different projects and compiling others' libraries during your own build process may introduce unnecessary complexity and project dependencies.
Some libraries require you to have some tools and plug-ins installed in your environment.
Instead of including the open source code in the build process, usually I have a dependencies or lib directory in the root of the Visual Studio solution with both the binaries and the source code. When I want to debug third-party code I just open another instance of Visual Studio and I attach there the debugger to the whole process.
Think some third-party open source libraries are very fat and maybe your own code base is small: since you add a lot of source code, your build process will delay a lot of because you just added useless stuff (code, plug-ins....).
I prefer the principle of delegating the build process to the author of the library and, in my case, I see myself as a consumer of such library: I don't need to recompile it everytime.
Ah, and if you're planning to modify the source code, the best practice should be contacting the author and sending to him/her the patches so he/she can include it in the official distribution. If not, you know: everytime the author releases a new version, you'll have the extra job of merging the new source code with your own changes...