How is the skipping implemented in Spring Batch?

Question

I was wondering how I could determine in my ItemWriter, whether Spring Batch was currently in chunk-processing-mode or in the fallback single-item-processing-mo

Answer 1

The implementation of the skip mechanism can be found in the FaultTolerantChunkProcessor and in the RetryTemplate.

Let's assume you configured skippable exceptions but no retryable exceptions. And there is a failing item in your current chunk causing an exception.

Now, first of all the whole chunk shall be written. In the processor's write() method you can see, that a RetryTemplate is called. It also gets two references to a RetryCallback and a RecoveryCallback.

Switch over to the RetryTemplate. Find the following method:

protected <T> T doExecute(RetryCallback<T> retryCallback, RecoveryCallback<T> recoveryCallback, RetryState state)

There you can see that the RetryTemplate is retried as long as it's not exhausted (i.e. exactly once in our configuration). Such a retry will be caused by a retryable exception. Non-retryable exceptions will immediately abort the retry mechanism here.

After the retries are exhausted or aborted, the RecoveryCallback will be called:

e = handleRetryExhausted(recoveryCallback, context, state);

That's where the single-item-processing mode will kick-in now!

The RecoveryCallback (which was defined in the processor's write() method!) will put a lock on the input chunk (inputs.setBusy(true)) and run its scan() method. There you can see, that a single item is taken from the chunk:

List<O> items = Collections.singletonList(outputIterator.next());

If this single item can be processed by the ItemWriter correctly, than the chunk will be finished and the ChunkOrientedTasklet will run another chunk (for the next single items). This will cause a regular call to the RetryCallback, but since the chunk has been locked by the RecoveryTemplate, the scan() method will be called immediately:

if (!inputs.isBusy()) {
    // ...
}
else {
    scan(contribution, inputs, outputs, chunkMonitor);
}

So another single item will be processed and this is repeated, until the original chunk has been processed item-by-item:

if (outputs.isEmpty()) {
    inputs.setBusy(false);

That's it. I hope you found this helpful. And I even more hope that you could find this easily via a search engine and didn't waste too much time, finding this out by yourself. ;-)

Answer 2

A possible approach to my original problem (the ItemWriter would like to know, whether it's in chunk or single-item mode) could be one of the following alternatives:

---

• Only when the passed chunk is of size one, any further checks have to be done

• When the passed chunk is a java.util.Collections.SingletonList, we would be quite sure, since the FaultTolerantChunkProcessor does the following:

  List items = Collections.singletonList(outputIterator.next());

  Unfortunately, this class is private and so we can't check it with instanceOf.

• In reverse, if the chunk is an ArrayList we could also be quite sure, since the Spring Batch's Chunk class uses it:

  private List items = new ArrayList();

• One blurring left would be buffered items read from the execution context. But I'd expect those to be ArrayLists also.

Anyway, I still find this method too vague. I'd rather like to have this information provided by the framework.

---

An alternative would be to hook my ItemWriter in the framework execution. Maybe ItemWriteListener.onWriteError() is appropriate.

Update: The onWriteError() method will not be called if you're in single-item mode and throw an exception in the ItemWriter. I think that's a bug a filed it: https://jira.springsource.org/browse/BATCH-2027

So this alternative drops out.

---

Here's a snippet to do the same without any framework means directly in the writer

    private int writeErrorCount = 0;

@Override
public void write(final List<? extends Long> items) throws Exception {
    try {
        writeWhatever(items);
    } catch (final Exception e) {
        if (this.writeErrorCount == 0) {
            this.writeErrorCount = items.size();
        } else {
            this.writeErrorCount--;
        }

        throw e;
    }
    this.writeErrorCount--;
}

public boolean isWriterInSingleItemMode() {
    return writeErrorCount != 0;
}

Attention: One should rather check for the skippable exceptions here and not for Exception in general.