Is there a way to create out of DOM elements in Web Worker?

Question

Context: I have a web application that processes and shows huge log files. They\'re usually only about 100k lines long, but it can be up to 4 million lines

Answer 1

No you can't create DOM elements in a web worker, but you can create a function that accepts the post message from that web worker, that does create the DOM elements. I think the deign that your looking for is called array chucking. And you would need to mix that with the web worker design pattern.

Answer 2

So is there any way, that I'm not aware of, to create DOM elements, outside the DOM tree, in a Web Worker?

No.

Why not? It seems to me that this can't create concurrency problems, as creating the elements could happen in parallel without problems.

Not for creating them, you're right. But for appending them to the main document - they would need to be sent to a different memory (like it's possible for blobs) so that they're inaccessible from the worker thereafter. However, there's absolutely no Document handling available in WebWorkers.

I create a DOM element for each line as soon as the data arrives (in JSON via ajax). Afterwards I save the elements in an array and I only show the lines that are visible.

Constructing over 500k DOM elements is the heavy task. Try to create DOM elements only for the lines that are visible. To improve performance and showing the first few lines faster, you also might chunk their processing into smaller units and use timeouts in between. See How to stop intense Javascript loop from freezing the browser

Answer 3

You have a couple of anti-patterns in your design:

1. Creating a DOM object has considerable overhead, and you are creating potentially millions of them at once.
2. Trying to get a web worker to manage the DOM is exactly what web workers are not for. They do everything else so the DOM event loop stays responsive.

You can use a cursor pattern to scroll through arbitrarily large sets of data.

1. DOM posts a message to worker with start position and number of lines requested (cursor).
2. Web worker random accesses logs, posts back the fetched lines (cursor data).
3. DOM updates an element with the async cursor response event.

This way, the heavy lifting is done by the worker, whose event loop is blocked during the fetch instead of the DOM, resulting in happy non-blocked users marvelling at how smooth all your animations are.

Answer 4

So you can't directly create DOM in a webworker - however, there may be another option to do a fair bit of your processing outside the main thread.

Check out this jsPerf I just created: http://jsperf.com/dom-construction-obj-vs-str

Essentially, you could be emitting POJSO's that have all the same values you get from a DOM, and convert it to DOM objects after receiving the message (this is what you're doing when you get HTML back, after all; POJSOs are just lower overhead, by virtue of not requiring further string processing). In this way you could even do things like emit event listeners and such (by, say, prefixing the event name with '!', and having the value map to some template-supplied view argument).

Meanwhile, without the DOM parser available, you'll need your own thing to convert a template as-needed, or to compile one to a format that's fast.

Answer 5

I don't see any reason why you can't construct html strings using web-workers. But I also don't think there would be much of a performance boost.

This isn't related to Web-Workers, but it relates to the problem you're trying to solve. Here are some thing that might help speed things up:

1. Use DocumentFragments. Add elements to them as the data comes in, and add the fragments to the DOM at an interval (like once a second). This way you don't have to touch the DOM (and incur a redraw) every time a line of text is loaded.

2. Do loading in the background, and only parse the lines as the user hits the bottom of the scroll area.

Answer 6

You have to understand the nature of a webworker. Programming with threads is hard, especially if you're sharing memory; weird things can happen. JavaScript is not equipped to deal with any kind of thread-like interleaving.

The approach of webworkers is that there is no shared memory. This obviously leads to the conclusion that you can't access the DOM.