org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step

Question

I am learning spring batch and I was able to create simple single step application(github repo link)

This application contains a job which does following:
1. rea

Answer 1

Actually, i don't have used the partioner myself, but i might be able to give you some hints.

The first exception you had (Stream closed) was due to the fact that the same reader instance was used for every slave process. Obviously, the first slave process to finish closes the reader and from that moment on, the other slave processes tried to read from a closed stream.

You fixed that with the StepScope-Annotation, which was the right approach to solve this problem.

The problem with the partioner approach is, that you are responsible to partition the data you read.

What you did was simply to create a partioner which had 20 slave processes and everyone of these slave processes read the whole file. Hence, your db contained every entry in the file 20 times.

What you should do is to configure every step instance with appropriate "context-properties". Based on these context properties, the step instance knows which lines it should process (for instance start and end line). Or you could also split up the original file into 20 files with different names and provide for every instance another filename.

I found two examples where this is explained, one where you read from a db and one that reads from files:

https://www.mkyong.com/spring-batch/spring-batch-partitioning-example/

https://www.baeldung.com/spring-batch-partitioner