Iterate over large collection in MongoDB via spring-data

Question

Friends!

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all el

Answer 1

you can still use mongoTemplate to access the Collection and simply use DBCursor:

     DBCollection collection = mongoTemplate.getCollection("boundary");
     DBCursor cursor = collection.find();        
     while(cursor.hasNext()){
         DBObject obj = cursor.next();
         Object object =  obj.get("polygons");
         ..
      ...
     }

Answer 2

You may want to try the DBCursor way like this:

    DBObject query = new BasicDBObject(); //setup the query criteria
    query.put("method", method);
    query.put("ctime", (new BasicDBObject("$gte", bTime)).append("$lt", eTime));

    logger.debug("query: {}", query);

    DBObject fields = new BasicDBObject(); //only get the needed fields.
    fields.put("_id", 0);
    fields.put("uId", 1);
    fields.put("ctime", 1);

    DBCursor dbCursor = mongoTemplate.getCollection("collectionName").find(query, fields);

    while (dbCursor.hasNext()){
        DBObject object = dbCursor.next();
        logger.debug("object: {}", object);
        //do something.
    }

Answer 3

Streams as cursor:

@Query("{}")
Stream<Alarm>  findAllByCustomQueryAndStream();

So for the large amount of data you can stream them and process the line by line without memory limitation

Answer 4

Use MongoTemplate::stream() as probably the most appropriate Java wrapper to DBCursor

Answer 5

Check new method to handle results per document basis.

http://docs.spring.io/spring-data/mongodb/docs/current/api/org/springframework/data/mongodb/core/MongoTemplate.html#executeQuery-org.springframework.data.mongodb.core.query.Query-java.lang.String-org.springframework.data.mongodb.core.DocumentCallbackHandler-

Answer 6

Late response, but maybe will help someone in the future. Spring data doesn't provide any API to wrap Mongo DB Cursor capabilities. It uses it within find methods, but always returns completed list of objects. Options are to use Mongo API directly or to use Spring Data Paging API, something like that:

        final int pageLimit = 300;
        int pageNumber = 0;
        Page<T> page = repository.findAll(new PageRequest(pageNumber, pageLimit));
        while (page.hasNextPage()) {
            processPageContent(page.getContent());
            page = repository.findAll(new PageRequest(++pageNumber, pageLimit));
        }
        // process last page
        processPageContent(page.getContent());

UPD (!) This method is not sufficient for large sets of data (see @Shawn Bush comments) Please use Mongo API directly for such cases.