couchdb views os_process_error - big documents

Question

I have a database that we're filling up with documents, some of which are 10 KB, and some of which that are ~70 MB in size.

Any view that tries to load these large documents fails with error: {"error":"os_process_error","reason":"{exit_status,0}"}, even this elementary one:

function(doc) {
    emit(null, "test");
}

os_process_error is set to 35000, but calling the view dies in ~15 seconds:

time curl -X GET http://localhost:5984/portal_production/_design/all_data_keys/_view/view1
{"error":"os_process_error","reason":"{exit_status,0}"}

real    0m15.907s
user    0m0.016s
sys 0m0.000s

From a few other threads, seems like this is related stack size: couchdb issue thread. I've applied this patch to increase the stack size, although I couldn't find a reference to the method's value. So I'm shooting in the dark a bit.

Soooo.... the real question is: what is the correct approach for using views on large documents? I'm sure I'm not the first to have this problem.

Answer 1

You'd hit couchjs stack size limit. If you're using CouchDB 1.4.0+ his size is limited by 64 MiB by default. You may increase it by specifying -S <number-of-bytes> option for JavaScript query server in CouchDB config. For instance, to set stack size to 128 MiB your config value will looks like:

[query_servers]
javascript = /usr/bin/couchjs -S 134217728 /usr/share/couchdb/server/main.js

Note, that /usr/bin/couchjs may be different for you depending on your OS. After adding this changes you need to restart CouchDB.

If you'll try to update JavaScript query server config though HTTP API, make sure to kill all couchjs processes from shell to let them apply changes.

If your CouchDB version is <1.4 try to upgrade first.