This is a case where you have lots of choice.
GeoTools is indeed a low level library and an excellent choice for some hands on work. We recently started using Swing for our example code in order to make tutorials more visual. That said they are tutorial code only and do not represent a multi threaded system such as uDig.
If you are looking for Swing applications to start form there is the Jump family (sadly scattered through a closed development process - OpenJump represents a collection of the various forks), gvSig which has excellent financial backing in Span, and so on.
You may also wish to look at the library side of the deegree project (which I seem to recall produced a desktop application as well?).
I also recommend looking at some of the tiled map viewers produced around Nasa World Wind and Open Street Map datasets. These will be more similar to the OpenLayers experience.
Disclaimer I am on the steering committee for GeoTools, and uDig.