Firstly I want mention that I have already read all articles on SO on subject, but still have no answer for my question. Also my question slightly different than others.
Yes, this can be (and undoubtedly is) done by hooking WM_NCPAINT and painting their button in response to it. The "glowing" is just a matter of picking the appropriate color and shading (and possibly doing a bit of alpha blending to have some "glow" show up close to the button).
In Vista and Windows 7 there is a new thing called the Desktop Window Manager. This is used to draw the "Aero glass" window titlebars, and do the glow effects. The old Windows XP approach of implementing WM_NCPAINT handlers doesn't work with this new system, so you have to use a whole new API.
I'm sure I've seen some articles on doing what you're asking about, but can't find them right now. Some pages that might give you some leads are:
http://msdn.microsoft.com/en-us/magazine/cc163435.aspx
http://msdn.microsoft.com/en-us/library/aa969540(v=VS.85).aspx
http://delphihaven.wordpress.com/2010/04/19/setting-up-a-custom-titlebar/