Installing third-party components always take a long time specially if you have large ones, but also it take more time if you setup the environment in more than one computer.
If we have the source, then we include that in our repository, under a separate folder.
If we don't have the source, then we just keep the most recent binaries (bpl, dll, whatever) in the repository, and include installation / usage instructions in a setup document.
It looks like this:
\root
\third_party_stuff
\vendor1 --we *do* have the source for this
\src
\bin
\vendor2 --we *do* have the source for this
\src
\bin
\vendor3 --we don't have the source for this one
\bin
\our_stuff
\project1
\src
\bin
\project2
\src
\bin