Is it safe to fork from within a thread?

Question

Let me explain: I have already been developing an application on Linux which forks and execs an external binary and waits for it to finish. Results are communicated by shm f

Answer 1

Provided you quickly either call exec() or _exit() in the forked child process, you're ok in practice.

You might want to use posix_spawn() instead which will probably do the Right Thing.

Answer 2

forking, even with threads, is safe. Once you fork, the threads are independent per process. (That is, threading is orthogonal to forking). However, if the threads in different processes use the same shared memory to comunicate, you have to devise a synchronization mechanism.

Answer 3

If you are using the unix 'fork()' system call, then you are not technically using threads- you are using processes- they will have their own memory space, and therefore cannot interfere with eachother.

As long as each process uses different files, there should not be any issue.

Answer 4

My experience of fork()'ing within threads is really bad. The software generally fails pretty quickly.

I've found several solutions to the matter, although you may not like them much, I think these are generally the best way to avoid close to undebuggable errors.

1. Fork first

   Assuming you know the number of external processes you need at the start, you can create them upfront and just have them sit there waiting for an event (i.e. read from a blocking pipe, wait on a semaphore, etc.)

   Once you forked enough children you are free to use threads and communicate with those forked processes via your pipes, semaphores, etc. From the time you create a first thread, you cannot call fork anymore. Keep in mind that if you're using 3rd party libraries which may create threads, those have to be used/initialized after the fork() calls happened.

   Note that you can then start using threads within the main and fork()'ed processes.

2. Know your state

   In some circumstances, it may be possible for you to stop all of your threads to start a process and then restart your threads. This is somewhat similar to point (1) in the sense that you do not want threads running at the time you call fork(), although it requires a way for you to know about all the threads currently running in your software (something not always possible with 3rd party libraries).

   Remember that "stopping a thread" using a wait is not going to work. Such a wait require a mutex and those need to be unlocked when you call fork(). You just cannot know when the wait is going to unlock/re-lock the mutex.

3. Choose one or the other

   The other obvious possibility is to choose one or the other and not bother with whether you're going to interfere with one or the other. This is by far the simplest method if at all possible in your software.

In my programming, I used all three solutions. I used Point (2) because the threaded version of log4cplus and I needed to use fork() for some parts of my software.

As mentioned by others, if you are using a fork() to then call execve() then the idea is to use as little as possible between the two calls. That is likely to work 99.999% of the time (many people use system() or popen() with fairly good successes too and these do similar things).

On the other, if like me you want to do a fork() and never call execve(), then it's not likely to work right while any thread is running.

Answer 5

Back at the Dawn of Time, we called threads "lightweight processes" because while they act a lot like processes, they're not identical. The biggest distinction is that threads by definition live in the same address space of one process. This has advantages: switching from thread to thread is fast, they inherently share memory so inter-thread communications are fast, and creating and disposing of threads is fast.

The distinction here is with "heavyweight processes", which are complete address spaces. A new heavyweight process is created by fork(2). As virtual memory came into the UNIX world, that was augmented with vfork(2) and some others.

A fork(2) copies the entire address space of the process, including all the registers, and puts that process under the control of the operating system scheduler; the next time the scheduler comes around, the instruction counter picks up at the next instruction -- the forked child process is a clone of the parent. (If you want to run another program, say because you're writing a shell, you follow the fork with an exec(2) call, which loads that new address space with a new program, replacing the one that was cloned.)

Basically, your answer is buried in that explanation: when you have a process with many LWPs threads and you fork the process, you will have two independent processes with many threads, running concurrently.

This trick is even useful: in many programs, you have a parent process that may have many threads, some of which fork new child processes. (For example, an HTTP server might do that: each connection to port 80 is handled by a thread, and then a child process for something like a CGI program could be forked; exec(2) would then be called to run the CGI program in place of the parent process close.)

Answer 6

While you can use Linux's NPTL pthreads(7) support for your program, threads are an awkward fit on Unix systems, as you've discovered with your fork(2) question.

Since fork(2) is a very cheap operation on modern systems, you might do better to just fork(2) your process when you have more handling to perform. It depends upon how much data you intend to move back and forth, the share-nothing philosophy of forked processes is good for reducing shared-data bugs but does mean you either need to create pipes to move data between processes or use shared memory (shmget(2) or shm_open(3)).

But if you choose to use threading, you can fork(2) a new process, with the following hints from the fork(2) manpage:

   *  The child process is created with a single thread — the
      one that called fork().  The entire virtual address space
      of the parent is replicated in the child, including the
      states of mutexes, condition variables, and other pthreads
      objects; the use of pthread_atfork(3) may be helpful for
      dealing with problems that this can cause.