pymongo - Message length is larger than server max message size

问题

the line for doc in collection.find({'is_timeline_valid': True}): is giving the Message Length error. How can I get all the collection without the error? I know about the find().limit() but I don't know how to use it.

Code:

from openpyxl import load_workbook
import pymongo
import os

wb = load_workbook('concilia.xlsx')
ws = wb.active
client = pymongo.MongoClient('...')
db = client['...']
collection = db['...']

r = 2
for doc in collection.find({'is_timeline_valid': True}):
   for dic in doc['timeline']['datas']:
     if 'concilia' in dic['tramite'].lower():
        ws.cell(row = r, column = 1).value = doc['id_process_unformatted']
        ws.cell(row = r, column = 2).value = dic['data']
        ws.cell(row = r, column = 3).value = dic['tramite']
        wb.save('concilia.xlsx')
        print('*****************************')
        print(dic['tramite'])
        # print('check!')
        r += 1

回答1:

Here is a simple paginator that splits the query execution into paginated queries.

from itertools import count

class PaginatedCursor(object):
    def __init__(self, cur, limit=100):
        self.cur = cur
        self.limit = limit
        self.count = cur.count()

    def __iter__(self):
        skipper = count(start=0, step=self.limit)

        for skip in skipper:
            if skip >= self.count:
                break

            for document in self.cur.skip(skip).limit(self.limit):
                yield document

            self.cur.rewind()

...
cur = collection.find({'is_timeline_valid': True})
...
for doc in PaginatedCursor(cur, limit=100):
   ...

回答2:

I ran into this problem today, and it turns out it has to do with the size of a particular document within the collection exceeding the max_bson_size limit. When adding documents to the collection, make sure the document size doesn't exceed max_bson_size size.

document_size_limit = client.max_bson_size
assert len(json.dumps(data)) < document_size_limit

I'm currently investigating why the collection allowed a document larger than max_bson_size in the first place.

回答3:

we can add batch_size to find() to reduce the message size.

for doc in collection.find({'is_timeline_valid': True}):

becomes

for doc in collection.find({'is_timeline_valid': True}, batch_size=1):

来源：https://stackoverflow.com/questions/49014697/pymongo-message-length-is-larger-than-server-max-message-size

标签

python

mongodb

pymongo