I have a psychological tic which makes me reluctant to use large libraries (like GLib or Boost) in lower-level languages like C and C++. In my mind, I think:
FFTW and ATLAS are two quite large libraries. Oddly enough, they play large roles in the fastest software in the world, applications optimized to run on supercomputers. No, using large libraries doesn't make your code slow, especially when the alternative is implementing FFT or BLAS routines for yourself.