CouchDB: Return Newest Documents of Type Based on Timestamp

问题

I have a system that accepts status updates from a variety of unique sources, and each status update creates a new document in the following structure:

{
 "type": "status_update",
 "source_id": "truck1231",
 "timestamp": 13023123123,
 "location": "Boise, ID"
}

Data purely example, but gets the idea across.

Now, these documents are generated at interval, once an hour or so. An hour later, we might the insert:

{
 "type": "status_update",
 "source_id": "truck1231",
 "timestamp": 13023126723,
 "location": "Madison, WI"
}

All I'm interested in doing is seeing the latest update from each unique source. I'm currently doing this by taking a map of:

function(doc) {
  if (doc.type == "status_update") {
    emit(doc.source_id, doc);
  }
}

And a reduce of:

function(keys, values, rereduce) {
  var winner = values[0];
  var i = values.length;
  while (i--) {
    var val = values[i];
    if (val.timestamp > winner.timestamp) winner = val;
  }
  return winner;
}

And querying the data as a reduce with group=true. This works as expected and provides a keyed result of only the latest updates.

The problem is it's terribly slow and requires me to reduce_limit=false in the CouchDB config.

It feels like there has to be a more efficient way of doing this. Updating the same document is not an option-- the history is important even though I don't require it in this instance. Processing the data client-side isn't an option either as this is a CouchApp and the amount of documents in the system is actually quite large and not practical to send them all over the wire.

Thanks in advance.

回答1:

You can get the latest timestamp for every source using the _stats built-in reduce function, then do another query to get the documents. Here's the views:

"views": {
  "latest_update": {
    "map": "function(doc) { if (doc.type == 'status_update') emit(doc.source_id, doc.timestamp); }",
    "reduce": "_stats"
  },
  "status_update": {
    "map": "function(doc) { if (doc.type == 'status_update') emit([doc.source_id, doc.timestamp], 1); }"
  }
}

First query latest_update with group=true, then status_update with something like (properly url-encoded):

keys=[["truck123",TS123],["truck234",TS234],...]&include_docs=true

where TS123, and TS234 are the values of max returned by latest_update.

回答2:

CouchDB map/reduce is incremental which basically means the results are always cached, so subsequent requests for the same view (even with different search parameters) run "for free" (or in logarithmic time).

However, that is not strictly true with reduce groups. Sometimes partial results must be re-reduced on the fly. Maybe that is what you are hitting.

Instead, how about a map view (i.e. no reduce function) that emits rows like this, with an array as the key:

// Row diagram (pseudo-code, just to show the concept).
// Key                    , Value
// [source_id, timestamp] , null // value is not very important in this example
["truck1231", 13023123123], null
["truck1231", 13023126723], null
["truck5555", 13023126123], null
["truck6666", 13023000000], null

Notice how all timestamps for a source "clump" together. (Actually, they collate.) To find the latest timestamp for "truck1231", just requests the last row in that "clump". To do that, do a descending query, from the end, with a limit=1 argument. To specify the "end", use the {} "high key" value as the second element in the key (see the collation link for details).

?descending=true&limit=1&startkey=["truck1231",{}]

(Actually, since your timestamps are integers, you could emit their negation, e.g. -13023123123. That will simplify your query a bit but—I don't know—that seems like playing with fire to me.)

To produce these kinds of rows, us a map function like this:

function(doc) {
  // Emit rows sorted first by source id, and second by timestamp
  if (doc.type == "status_update" && doc.timestamp) {
    emit([doc.source_id, doc.timestamp], null) // Using `doc` as the value would be fine too
  }
}

回答3:

I suspect that it's slow only because you emit the entire document, which means a lot of data needs to be stored and moved around to compute your final values. Try emitting the timestamp instead:

function(doc) {
  if (doc.type == "status_update") {
    emit(doc.source_id, [doc._id,doc.timestamp]);
  }
}

function(keys, values, rereduce) {
  var winner = values[0];
  var i = values.length;
  while (i--) {
    var val = values[i];
    if (val[1] > winner[1]) winner = val;
  }
  return winner;
}

This should get you an [id,timestamp] pair for every key without being too slow or having to store too much data in the views.

Once you have a list of identifiers on the client, send a second request using the bulk GET API:

_all_docs?keys=[id1,id2,id3,...,idn]&include_docs=true

This will grab all the documents in one request.

来源：https://stackoverflow.com/questions/10389110/couchdb-return-newest-documents-of-type-based-on-timestamp

标签

couchdb

MapReduce

couchapp