Lua :: How to write simple program that will load multiple CPUs?

Question

I haven\'t been able to write a program in Lua that will load more than one CPU. Since Lua supports the concept via coroutines, I believe it\'s achievable.

Reason for me

Answer 1

It is not possible with the core Lua libraries (if you don't count creating multiple processes and communicating via input/output), but I think there are Lua bindings for different threading libraries out there.

The answer from jpjacobs to one of the related questions links to LuaLanes, which seems to be a multi-threading library. (I have no experience, though.)

If you embed Lua in an application, you will usually want to have the multithreading somehow linked to your applications multithreading.

Answer 2

In addition to already suggested LuaLanes, llthreads and other stuff mentioned here, there is a simpler way.

If you're on POSIX system, try doing it in old-fashioned way with posix.fork() (from luaposix). You know, split the task to batches, fork the same number of processes as the number of cores, crunch the numbers, collate results.

Also, make sure that you're using LuaJIT 2 to get the max speed.

Answer 3

It's very easy just create multiple Lua interpreters and run lua programs inside all of them.

Lua multithreading is a shared nothing model. If you need to exchange data you must serialize the data into strings and pass them from one interpreter to the other with either a c extension or sockets or any kind of IPC.

Answer 4

Serializing data via IPC-like transport mechanisms is not the only way to share data across threads.

If you're programming in an object-oriented language like C++ then it's quite possible for multiple threads to access shared objects across threads via object pointers, it's just not safe to do so, unless you provide some kind of guarantee that no two threads will attempt to simultaneously read and write to the same data.

There are many options for how you might do that, lock-free and wait-free mechanisms are becoming increasingly popular.

Answer 5

Lua coroutines are not the same thing as threads in the operating system sense.

OS threads are preemptive. That means that they will run at arbitrary times, stealing timeslices as dictated by the OS. They will run on different processors if they are available. And they can run at the same time where possible.

Lua coroutines do not do this. Coroutines may have the type "thread", but there can only ever be a single coroutine active at once. A coroutine will run until the coroutine itself decides to stop running by issuing a coroutine.yield command. And once it yields, it will not run again until another routine issues a coroutine.resume command to that particular coroutine.

Lua coroutines provide cooperative multithreading, which is why they are called coroutines. They cooperate with each other. Only one thing runs at a time, and you only switch tasks when the tasks explicitly say to do so.

You might think that you could just create OS threads, create some coroutines in Lua, and then just resume each one in a different OS thread. This would work so long as each OS thread was executing code in a different Lua instance. The Lua API is reentrant; you are allowed to call into it from different OS threads, but only if are calling from different Lua instances. If you try to multithread through the same Lua instance, Lua will likely do unpleasant things.

All of the Lua threading modules that exist create alternate Lua instances for each thread. Lua-lltreads just makes an entirely new Lua instance for each thread; there is no API for thread-to-thread communication outside of copying parameters passed to the new thread. LuaLanes does provide some cross-connecting code.

Answer 6

In addition to LuaLanes, take a look at llthreads