Concurrent data structure design

Question

I am trying to come up with the best data structure for use in a high throughput C++ server. The data structure will be used to store anything from a few to several million obje

Answer 1

Personally, I'm quite fond of persistent immutable data structures in highly-concurrent situations. I don't know of any specifically for C++, but Rich Hickey has created some excellent (and blisteringly fast) immutable data structures in Java for Clojure. Specifically: vector, hashtable and hashset. They aren't too hard to port, so you may want to consider one of those.

To elaborate a bit more, persistent immutable data structures really solve a lot of problems associated with concurrency. Because the data structure itself is immutable, there isn't a problem with multiple threads reading/iterating concurrently (so long as it is a const iterator). "Writing" can also be asynchronous because it's not really writing to the existing structure but rather creating a new version of that structure which includes the new element. This operation is made efficient (O(1) in all of Hickey's structures) by the fact that you aren't actually copying everything. Each new version shares most of its structure with the old version. This makes things more memory efficient, as well as dramatically improving performance over the simple copy-on-write technique.

With immutable data structures, the only time where you actually need to synchronize is in actually writing to a reference cell. Since memory access is atomic, even this can usually be lock-free. The only caveat here is you might lose data between threads (race conditions). The data structure will never be corrupted due to concurrency, but that doesn't mean that inconsistent results are impossible in situations where two threads create a new version of the structure based on a single old and attempt to write their results (one of them will "win" and the other's changes will be lost). To solve this problem, you either need to have a lock for "writing operations", or use some sort of STM. I like the second approach for ease-of-use and throughput in low-collision systems (writes are ideally non-blocking and reads never block), but either one will work.

You've asked a tough question, one for which there isn't really a good answer. Concurrency-safe data structures are hard to write, particularly when they need to be mutable. Completely lock-free architectures are provably impossible in the presence of shared state, so you may want to give up on that requirement. The best you can do is minimize the locking required, hence the immutable data structures.

Answer 2

If you don't need a sort order, don't use a red/black tree or anything else that inherently sorts.

Your question is not well specified enough w.r.t to interaction between reads and writes. Would it be ok if a "read" is implemented by a lock+copy+unlock and then use the new copy?

You may want to read about seqlocks in http://en.wikipedia.org/wiki/Seqlock, and on "lock free" processes in general -- though, you may want to relax your requirements as much as possible -- a lock-free hash table implementation is a major undertaking.

Answer 3

I'm not sure if anybody has mentioned this, but I would take inspiration from Java's ConcurrentHashMap. It offers traversal, retrieval and insertion without locking or waiting. The only lock occurs once you've found a bucket of data corresponding to the hash key and you're traversing that bucket (i.e. you ONLY lock the bucket not the actual hash map). "Instead of a single collection lock, ConcurrentHashMap uses a fixed pool of locks that form a partition over the collection of buckets."

You can find more details on the actual implementation here. I believe that all of the things shown in the implementation can be just as easily done with C++.

So let's go through your list of requirements:

1. High throughput. CHECK
2. Thread safe. CHECK
3. Efficient inserts happen in O(1). CHECK
4. Efficient removal (with no data races or locks). CHECK
5. VERY efficient traversal. CHECK
6. Does not lock or wait. CHECK
7. Easy on the memory. CHECK
8. It is scalable (just increase the lock pool). CHECK

Here is an example of a map Entry:

protected static class Entry implements Map.Entry {
    protected final Object key;
    protected volatile Object value;
    protected final int hash;
    protected final Entry next;
    ...
}

Note that the value is volatile, so when we're removing an Entry we set the value to NULL which is automatically visible to and any other thread that attempts to read the value.

Answer 4

Well, to be thread-safe you're going to have to lock something at some point. One key thing is to make sure that the objects in your repository can be locked separately from the repository structure itself: i.e. don't have a _next link or anything of the sort inside the data you are storing. This way read operations can lock the contents of the objects without locking the structure of the repository.

Efficient insert is easy: linked list, unsorted arrays, hashtables all work ok. Efficient deletion is harder since that involves finding the deleted thing in the repository. Howerver, for raw simplicity and speed, a linked list is a good choice. Can deletions be postponed for non-busy times and items just marked as "inactive"? Then the cost of finding/deleting is not so limiting.

You're still going to have problems with traversal though. About all you can do is to lock and take a snapshot of what needs to be traversed, then check for any changes after the snapshot is looked at. Tough problem...

Answer 5

I am a bit late to the party. But if someone is still looking for a practical solution to this problem and they have not yet decided on a server, let me suggest Google's App Engine. Their Datastore is optimized for these type of requirements.

Answer 6

You have 3 types of tasks:

1. iteration (slow)
2. insertion (fast)
3. deletion (fast)

If near consistency is good enough then keep track of the # of active iteration tasks.

If iterations tasks are active and a new insert or deletion tasks comes in queue those tasks for later processing (but you can return the to caller right away)

As soon as the last iteration if finished process queued inserts and deletes.

If an iteration request comes in while inserts or deletes are pending then queue it up.

If an iteration request comes in while there are just iterations running just have it go and iterate.

You should still write the iteration to be as fast as possible by making a copy of the data you are iterating over and then process that data in the client if the actual data processing takes a lot more time than the iteration itself.

I would implement the main collection with a hashtable or stl:map might even be fast enough. Insert/Delete requests could be queued in a list.