Using tensorflow on a GPU but also numpy arrays in same code: memory overhead delay?

Question

I am calculating Fourier transforms with tensorflow using tf.signal.fft. I have successfully installed tensorflow-gpu and have the right drivers an