Document Databases: Redundant data, references, etc. (MongoDB specifically)

Question

It seems like I run into lots of situations where the appropriate way to build my data is to split it into two documents. Let's say it was for a chain of stores and you were saving which stores each customer had visited. Stores and Customers need to be independent pieces of data because they interact with plenty of other things, but we do need to relate them.

So the easy answer is to store the user's Id in the store document, or the store's Id in the user's document. Often times though, you want to access 1-2 other pieces of data for display purposes because Id's aren't useful. Like maybe the customer name, or the store name.

1. Do you typically store a duplicate of the entire document? Or just store the pieces of data you need? Maybe depends on the size of the doc vs how much of it you need.
2. How do you handle the fact that you have duplicate data? Do you go hunt down data when it changes? Update the data at some interval when it's loaded? Only duplicate when you can afford stale data?

Would appreciate your input and/or links to any kind of 'best practices' or at least well-reasoned discussion of these topics.

Answer 1

There are basically two scenario's: fresh and stale.

Fresh data

Storing duplicate data is easy. Maintaining the duplicate data is the hard part. So the easiest thing to do is to avoid maintenance, by simply not storing any duplicate data to begin with. This is mainly useful if you need fresh data. Only store the references, and query the collections when you need to retrieve information.

In this scenario, you'll have some overhead due to the extra queries. The alternative is to track all locations of duplicate data, and update all instances on each update. This also involves overhead, especially in N-to-M relations like the one you mentioned. So either way, you will have some overhead, if you require fresh data. You can't have the best of both worlds.

Stale data

If you can afford to have stale data, things get a lot easier. To avoid query overhead, you can store duplicate data. To avoid having to maintain duplicate data, you're not going to store duplicate data. At least not actively.

In this scenario you'll also want to store only the references between documents. Then use a periodic map-reduce job to generate the duplicate data. You can then query the single map-reduce result, rather than separate collections. This way you avoid the query overhead, but you also don't have to hunt down data changes.

Summary

Only store references to other documents. If you can afford stale data, use periodic map-reduce jobs to generate duplicate data. Avoid maintaining duplicate data; it's complex and error-prone.

Answer 2

The answer here really depends on how current you need your data to be.

@Niels has a good summary here, but I think it's fair to note that you can "cheat".

Let's say that you want to display the Stores used by a User. The obvious problem here is that you can't "embed" the Store inside the User b/c the Store is too important on its own. But what you can do is embed some Store data in the User.

Just use the stuff you want for display like "Store Name". So your User object would look like this:

{
  _id : MongoID(),
  name : "Testy Tester",
  stores : [ 
             { _id : MongoID(), "name" : 'Safeway' },
             { _id : MongoID(), "name" : 'Walmart' },
             { _id : MongoID(), "name" : 'Best Buy' }
            ]
}

This way you can display the typical "grid" view, but require a link to get more data about the store.

Answer 3

To answer your direct questions:

1. No duplicates.
2. No duplicates.

;)

The only duplicates you should ever have are "simple" values like weights (which may happen to be the same, but aren't any more efficient in either time or space to store separately), and ids referencing another object (which are duplicate values, but much smaller and more manageable than the duplicate object data they replace).

Now, to answer your scenario: what you want is a Many-to-Many relationship. The usual solution here is to make a third "through" or "bridge" table/collection, probably called StoreUsers:

StoreUsers
----------
storeuser_id
store_id
user_id

You add a record to this for each link between stores and users, whether it's for a different store, a different user, or a bunch of users in one store. You can then look this up independently, for either the Store, or the User. MongoDB advocates this approach too; it's not RDBMS-specific.