Every time I add a new form to my project, it drops a big glop of boilerplate in the uses clause.
uses
Windows, Messages, SysUtils, Variants, Classes, Grap
Actually, that bothered me aswell.
But then i checked the executable size compiled with the default units and then checked it without them (Messages, Variants, Grahpics) and the size was not much smaller.
Dialogs.pas adds some kilobytes, but I frequently use delphi VCL dialogs over Win API ones.
So, I'd say not bother too much with them