Can we have multiple readers(reading records through pagination) in a single batch job

问题

I have a spring batch job to be written . where :

1. I need to read say 10k records through pagination(fetching 1000 rec at a time) (from azure SQL db)
2. I need to use 1000 records at a time and then use one column of these records(say some id) to read corresponsing records from another cosmos db table.

How do I implement 2 readers in this case as I need to read 1000 records at a time and again fetch records from cosmos db for those 1000 records first and process that.

回答1:

There is a common pattern called the "driving query pattern" in which the reader reads only item IDs and a processor does another query to fetch item details.

This pattern works well for small/medium data sets but you should be aware that it does not perform well for large datasets as there would be an additional query for each item (also known as the n+1 problem).

来源：https://stackoverflow.com/questions/63979027/can-we-have-multiple-readersreading-records-through-pagination-in-a-single-bat