I'll use jQuery as an example, but what I'm saying here could apply to most JavaScript frameworks.
Many frameworks (notably jQuery) are far too monolithic and not modular enough.
While depending on well-tested 3rd party software is often more than justified, "frameworks" tend to give you a lot more functionality than you need at the moment.
In many projects, I very much like the convenience that jQuery gives me for selecting sets of elements (using $(".classname"), for example). But, if I'm not using any significant amount of AJAX, I don't need the AJAX utilities provided by jQuery.
Software should do one thing and do it well, and software written in JavaScript is no exception. Most of the frameworks you refer to try to do everything, resulting in unnecessary complexity.
One place this can bite you is when you're considering upgrading to the next version of the framework. That involves crawling through jQuery's changelogs for backwards-incompatible changes and searching your project for areas where that code is used. This can be quite a nightmare, especially if you don't necessarily have a comprehensive list of which jQuery features you use and which ones you don't.
Also, jQuery (and other frameworks) tends to cause developers to start depending on new features of jQuery without even thinking about it, making it harder to determine which features of jQuery your project uses and which it doesn't.
If you use a utility which does one thing, then you know exactly which features of that utility you're using. There's only one. (If you aren't using that utility at all, it's easy to determine. Such a determination would mean you could safely remove it from your project.)
I'm all for using well-tested 3rd party code. But if it tries to do too much, (that is, if it's a framework rather than a utility), you should probably look for an alternative. If it tries to do too much (like jQuery tries to do too much), then it's got some serious, foundational design flaws that will probably come back to bite you.
