As a grizzled CMS veteran I can say that Umbraco is no harder to set up and use than many other CMS solutions.
However much of whether you find it hard or easy depends largely on your previous experience with CMS and your expectations for what a CMS should provide out of the box.
I've worked mostly with larger CMSs:
- Microsoft CMS
- Immediacy
- Obtree
- Reef (anyone remember that one!)
- etc....
Against those it is no harder to use and is probably easier as it tends to get out of your way and lets you get on with building the functionality you require.
However if your expectations are more based around things like Wordpress, i.e. install and go but with more limited options, then it can be hard to start with (if you just fire it up without installing a website starter kit).
My recommendation is that if you are building a small site you take a look at the Creative website starter kit at our.umbraco.org. There are also many packages that you can install to make things easier or add specific functionality (including pre-built navigation controls and full blog solutions).
Also take a look at the Wiki on our.umbraco.org and ask questions in the forum, the community is helpful and friendly.