How to override Spring Batch CompositeItemWriter manage transaction for delegate writers in case of exception arises?

Question

I am extending this How does Spring Batch CompositeItemWriter manage transaction for delegate writers? question here:

In my case I\'ve a below CompositeItemWrit

Answer 1

The main root cause of the issue was, we were trying to use two different ItemWriter to write the data into the same table and this was causing the Transaction to behave weirdly.

We've implemented SkipListenets (considering the fact that use may not often get the garbage or junk data as we're performing the validation at the initial data load.)

Since we’ve implemented “Spring Batch Skip Technique” in the Batch jobs, this help us to specify certain exception types and a maximum no. of skipped items and whenever one of those skippable exception is thrown, batch job doesn’t fail but skip that particular item and goes onto the next item. Only when maximum no. of skipped items is reached, batch job will fail. We’ve use skip logic with the “Fault tolerance” features of Spring Batch are applied to items in chunk-oriented steps, not to the entire step.

Therefore if Item failed to write at one delegate then it will be considered failed for all other delegates (that item will not be passed to another delegates) and we're fine with it because we're capturing the details in the error log table and from there we can reprocess it as when needed.

Answer 2

I can't see a way you could align spring-batch's single transaction for writing the whole chunk as atomic vs your idea of keeping the atomicity to individual writers as long as you want skiplistener.

I am not sure if this is possible but may be you will be able to test it quickly. This is how the message carries the exception in some integration frameworks like camel from one processor to error handling flow.

• You item reader should return a EmployeeWrapper which contains employee record and has a field to store Exception.

• your CompositeItemWriter receives List<EmployeeWrapper> and composite writer has 5 writers instead of 4. And the 5th writer will do what your SkipListener would have done.

    List<ItemWriter<? super EmployeeWrapper>> employee = new ArrayList<>();
    employee.add(employeeWriter());
    employee.add(departmentWriter());
    employee.add(stockWriter());
    employee.add(purchaseWriter());
    employee.add(errorRecordWriter());

• Your first 4 individual writers never throw exception, instead mark it as processed but add the caught exception as attribute of EmployeeWrapper.

• Your 5th errorRecordWriter receives all the records, check any record that has exception attribute added and writes them to error table. Incase it failed to write error record, you can throw the exception and all 5 writers will be retried.

• Regarding how you would know which record is error record when the batch update fails. It seems when an error occur in chunk, spring rollbacks the chunk and start retrying it record by record in that chunk so it knows which record is the problematic. So you can do the same thing in your individual writers. I.e catch the batch update exception and then retry them one by one to separate the error records

Answer 3

A couple things here:

1. Do not use @Transactional with Spring Batch - Spring Batch manages the transactions for you so using that annotation will cause issues. Do not use it.
2. Manage the exceptions yourself - In the scenario you are describing, where you want to call four ItemWriter implementations for the same item, but want to skip the exceptions at the delegated ItemWriter level, you will need to write your own CompositeItemWriter implementation. Spring Batch provides that level of composition (where we delegate to each ItemWriter implementation with the same item) out of convenience, but from the framework's perspective it is just a single ItemWriter. In order to handle exceptions at the child ItemWriter level, you will need to write your own wrapper and manage the exceptions yourself.

UPDATE:
An example implementation of the custom ItemWriter I'm referring to (note the code below is untested):

public class MyCompositeItemWriter<T> implements ItemWriter<T> {
      private List<ItemWriter<? super T>> delegates;
 
    @Override
      public void write(List<? extends T> items) throws Exception {
            for(ItemWriter delegate : delegates) {
               try {
                  delegate.write(items);
               }
               catch (Exception e) {
                  // Do logging/error handling here
               }
            }
    }

    @Override
    public void setDelegates(List<ItemWriter<? super T>> delegates) {
        super.setDelegates(delegates);
        this.delegates = delegates;
    }
}