I've used IDEs from the very beginning (arguably; having started with QBASIC), and for many, many years. I've now switched almost completely to VIM (in diverse flavours) for all my development work and I don't regret it. My productivtity has definitely increased.
Of course, nothing will replace the Windows Forms designer from Visual Studio. But compared to VIM (and Emacs, I'm sure) the text editor inside Visual Studio is really lousy. Once you harness the raw power of the console and the GNU developer tools (by which I mean make, GCC, binutils and gdb, and then some) you'll notice that these tools may look primitive but they're just the opposite, and actually offer all the tools that an IDE provides (well, except for the forms designer).
It's just that you've got a very steep climb ahead of you when you first start using these tools and the incentive may be small. I was lucky enough (?) to be forced to use these tools so I didn't have a choice that I could weasel out of.
