I recently got a mail forwarded that Sun is working on modernizing the Javadoc HTML output. From said mail:
We are proposing improvements to javadoc/doclet for JDK7. The
project wiki page is located at
http://wikis.sun.com/display/Javadoc/Home. As a part of the proposed
improvements, the UI of the javadoc output will be revamped. The new
design screenshots are uploaded to the project wiki. The javadoc output
markup will be modified to be valid HTML and WCAG 2.0 compliant.
So there is definitely still work going on there, even if somewhat late. However, in my eyes one of the biggest drawbacks of Javadoc is its very close coupling with HTML. Many classes have Javadoc which includes literal HTML and relies on the output being HTML, too. Unfortunate, but this won't change anytime, I think. Still, this means that developers are free to include whatever they want in HTML there which might as well be invalid, non-well-formed, etc. So adapting the output from the javadoc tool is only one part of this, the other won't and can't change and thus remains.
As for browsing documentation I also find the HTML documentation a little unwieldy. I usually use the Javadoc view in Eclipse. It has drawbacks as well (slow and you can't really search) but it's Good Enough™ for most things.