I\'m getting to the point on an app where I need to start caching things, and it got me thinking...
I could be way off here, but...
Have you ever looked at CouchDB? (I have no affiliation w/ them BTW) I could be way wrong, but your situation sounds like it may be a perfect fit for the use of the Apache CouchDB I haven't really used it yet myself but I took a good look at it a short while back and it is a very interesting database.
It is a document based database that uses a REST api for connections (very versatile and easy to use). It is also very JSON centric, very fast and a tiny footprint; they say it can reside on phones and other embedded uses too but at the same time is supposed to be extremely scalable (upwards that is). If your a big JS user (which you sound like you are) then you may be right at home with it.
I was just thinking that it may come in handy in any number of ways that have been proposed here and thought I'd chime in just to give you an idea for storage options :)
I used to mix these two approaches but then switched to client side rendering entirely because it was really hard to handle heavy JavaScript properly the otherwise. As a complete solution can recommend the approach of the JavaScriptMVC framework.
It has a view rendering engine called EJS which can compress your views into plain JavaScript for a production build of your application. That makes it extremely fast (all your HTML gets preloaded with your single compressed JavaScript file so it is rendered as soon as you receive your models JSON data). I personally couldn't notice a performance difference between EJS rendering and transferring plain HTML.
Then for your API, following the REST principles, caching is one of the key constraints. So if your application supports HTTP caching properly for JSON data and using your compressed client side templates you might even see a performance improvement.
Persistant private data storage.
You need a server to store data with various levels of public/private access. You also need a server for secure closed source information. You need a server to do heavy lifting that you don't want to do on the client. Complex data querying is best left upto your database engine. Indexing and searching is not yet optimised for javascript.
Also you have the issues of older browsers being far slower. If your not running FF4/Chrome or IE9 then there is a big difference between data manipulation and page construction on the client and the server.
I myself am going to be trying to build a web application made entirely using a MVC framework and template's but still using the server to connect to secure and optimised database.
But in general the application can indeed be build entirely in javascript and using templates. The various constructs and javascript engines have advanced enough to do this. There are enough popular frameworks out there to do this. The Pure javascript web applications are no longer experiments and prototypes.
Oh, and if were recommending frameworks for this, then take a look at backbone.js.
Security
Let's not forget that we do not trust the client. We need serverside validation. JavaScript is interpreted, dynamic and can be manipulated at run time. We never trust client input.