File segment/section/record locks in Linux threads

Question

I have a multi-threaded process where a file is shared (read and written) by multiple threads. Is there any way a thread can lock one file segment so that other threads cannot a

Answer 1

If you don't need file locks between different processes, avoid the file locks (which are one of the worst designed parts of the POSIX API) and just use mutexes or other shared memory concurrency primitives.

Answer 2

There are 2 ways you can do it:

1. Use Mutex to get a record's lock in a thread within the same process. Once the lock is acquired, any other thread in the process, mapping the file that tries to acquire the lock is blocked until the lock is released.(Preferable and only most straightforward solution available in Linux).

2. Semaphores and mutexes on a shared memory or a memory mapped file.

Answer 3

No. The region-locking feature you're asking about has surprising semantics and it is not being further developed because it is controlled by POSIX. (In fact, it is Kirk McKusick's preferred example of what's wrong with POSIX.) If there is a non-POSIX byte-range lock facility in Linux, I can't find it.

There is discussion of the problems of POSIX byte-range locking in a multithreaded world here: http://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html.

However, if you're concerned only with threads within one process, you can build your own region-locking using semaphores. For example:

#include <stdbool.h>
#include <pthread.h>
#include <sys/types.h>

// A record indicating an active lock.
struct threadlock {
  int fd;  // or -1 for unused entries.
  off_t start;
  off_t length;
};

// A table of all active locks (and the unused entries).
static struct threadlock all_locks[100];

// Mutex housekeeping.
static pthread_mutex_t mutex;
static pthread_cond_t some_lock_released;
static pthread_once_t once_control = PTHREAD_ONCE_INIT;
static void threadlock_init(void) {
  for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i)
    all_locks[i].fd = -1;
  pthread_mutex_init(&mutex, (pthread_mutexattr_t *)0);
  pthread_cond_init(&some_lock_released, (pthread_condattr_t *)0);
}

// True iff the given region overlaps one that is already locked.
static bool region_overlaps_lock(int fd, off_t start, off_t length) {
  for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i) {
    const struct threadlock *t = &all_locks[i];
    if (t->fd == fd &&
        t->start < start + length &&
        start < t->start + t->length)
      return true;
  }
  return false;
}

// Returns a pointer to an unused entry, or NULL if there isn't one.
static struct threadlock *find_unused_entry(void) {
  for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i) {
    if (-1 == all_locks[i].fd)
      return &all_locks[i];
  }
  return 0;
}

// True iff the lock table is full.
static inline bool too_many_locks(void) {
  return 0 == find_unused_entry();
}

// Wait until no thread has a lock for the given region
// [start, start+end) of the given file descriptor, and then lock
// the region. Keep the return value for threadunlock.
// Warning: if you open two file descriptors on the same file
// (including hard links to the same file), this function will fail
// to notice that they're the same file, and it will happily hand out
// two locks for the same region.
struct threadlock *threadlock(int fd, off_t start, off_t length) {
  pthread_once(&once_control, &threadlock_init);
  pthread_mutex_lock(&mutex);

  while (region_overlaps_lock(fd, start, length) || too_many_locks())
    pthread_cond_wait(&some_lock_released, &mutex);

  struct threadlock *newlock = find_unused_entry();
  newlock->fd = fd;
  newlock->start = start;
  newlock->length = length;

  pthread_mutex_unlock(&mutex);
  return newlock;
}

// Unlocks a region locked by threadlock.
void threadunlock(struct threadlock *what_threadlock_returned) {
  pthread_mutex_lock(&mutex);

  what_threadlock_returned->fd = -1;
  pthread_cond_broadcast(&some_lock_released);

  pthread_mutex_unlock(&mutex);
}

Caution: the code compiles but I haven't tested it even a little.

Answer 4

Yes - but not with the same mechanism. You'll have to use something like pthread mutexes, and keep track of the bookkeeping yourself.

Possible outline for how to make this work

• Wait on and claim a process-level mutex over a bookkeeping structure
  • make sure no other threads within your process are trying to use that segment
  • mark yourself as using the file segment

• Release the process-level mutex

• Grab fnctl lock for process (if necessary)

• Do your writing

• Release fnctl lock to allow other processes to use the segment (if necessary)

• Wait again on process-levelbookkeeping structure mutex (may not be necessary if you can mark it unused atomically)

  • mark segment as unused within your process.
• Release process-level mutex