How to process huge array of objects in nodejs

谁说我不能喝 提交于 2019-12-01 13:32:48

问题


I want to process array of length around 100 000, without putting too much load on CPU. I researched about streams and stumbled upon highlandjs, but i am unable to make it work.

I also tried using promises and processing in chunks but still it is putting very much load on CPU, program can be slow if needed but should not put load on CPU


回答1:


With node.js which runs your Javascript as single threaded, if you want your server to be maximally responsive to incoming requests, then you need to remove any CPU intensive code from the main http server process. This means doing CPU intensive work in some other process.

There are a bunch of different approaches to doing this:

  1. Use the child_process module to launch another nodejs app that is purposeful built for doing your CPU intensive work.
  2. Cluster your app so that you have N different processes that can do both CPU intensive work and handle requests.
  3. Create a work queue and a number of worker processes that will handle the CPU intensive work.
  4. Use the newer Worker Threads to move CPU intensive work to separate node.js threads (requires node v12+ for stable, non-experimental version of threads).

If you don't do this CPU intensive work very often, then #1 is probably simplest.

If you need scale for other reasons (like handling lots of incoming requests) and you don't do the CPU intensive stuff very often #2.

If you do the CPU intensive stuff pretty regularly and you want incoming request processing to always have the highest priority and you're willing to allow the CPU intensive stuff to take longer, then #3 (work queue) or #4 (threads) is probably the best and you can tune the number of workers to optimize your result.



来源:https://stackoverflow.com/questions/56917341/how-to-process-huge-array-of-objects-in-nodejs

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!