You're right: it's much nicer.
Is there an open-source library out there that provides [a document editor that doesn’t use the browser to handle editable text, instead using a custom editing surface and layout engine, entirely in JavaScript].
Doubtful.
First, the editor is dubbed "Kix". It's not non-HTML, but it doesn't use contentEditable because it sucks for what they're doing, as you mention. What it does is tricky and bulky to do right, and I'm sure it took them a good bit of effort to create it.
That said, why would Google open source Kix? It undermines their effort to be your documents hub, and it would mean they have to do a bunch of additional work to separate Kix from the rest of Google Docs in a way that doesn't leave it difficult to integrate. The benefit? Not much, seeing as though open source projects sponsored by Google are unlikely to see support in the form of developer-participation in improving it.
As for someone creating a library to do it, I'm doubtful it'll happen in open source. Usually open source projects that are a significant undertaking are university or company sponsored, and that's not likely in this case for the reasons above. Then again, if a developer has an itch for it, there's no telling where it could go…