Usually I compile against the binaries. I have a folder vendor/libs on my solutions, or I just use NuGet, and compile againest the Dlls.
But, sometimes I need to debug the open source library, when something is not happening as expected. And just then, I link with the source code (just for debugging).
When going to production, I use the binaries, because the code is already well tested.