Spring Batch - Skip Record On Process

Question

I wanted to skip some record on process.

what i have tried is, i have created custom exception and throw the exception when i want to skip the record and its calling

Answer 1

when we return null in the process() method, it will filter the record and increase the filter count.

@Transactional(propagation = Propagation.REQUIRED)
    @Override
    public SomeObject process(SomeObject someObject) throws Exception {
        if (some condition) {
            return null;
        }   
}

If We want to skip the record , throw an exception. This will skip the record and increase processSkipCount as well.

@Transactional(propagation = Propagation.REQUIRED)
    @Override
    public SomeObject process(SomeObject someObject) throws Exception {
        if (some condition) {
            throw new Exception("invalid record");
        }   
}

Add this Exception to context file as well.

<batch:skippable-exception-classes>
<batch:include class="java.lang.Exception" />
</batch:skippable-exception-classes>

Answer 2

There are indeed two ways to do this, one like you mention with skip mechanism and the other with returning null which will filter out item and not write it. Here is documentation link - 6.3.2. Filtering records where it is nicely explained what is difference between two approaches. Also this blog post explains skip in details and transactions in batch.

When you i.e. parse csv file and you expect 5 items per line but one line holds 6 items that is invalid item, and you can opt out to skip it (by marking reader exception as skippable and defining condition in policy as you gave example). However if each line holds name and your use case is to not write items that start with letter N that is better implemented with returning null (filtering item) since it is valid item but not according to your business case.

Please note also that if you return null number of those items will be in StepContext in getFilterCount() and if you use skip approach they will be in getReadSkipCount(), getProcessorSkipCount and getWriteSkipCount respectfully.

Answer 3

@Component
@Scope(value = "step")
public class XyzItemProcessor implements ItemProcessor<ABCInfo , ABCInfo > {

@Override
public ABCInfo process(ABCInfo abcInfo) throws Exception {

    if (abcInfo.getRecordType().equals("H") || extVoterInfo.getRecordType().equals("T"))
        return null;////this is how we skip particular record to persist in database
    else {
        return abcInfo;
    }
}
}

Return null will skip the particular record to be persisted in database

Answer 4

There are, as well, another way of not writing (skipping) something. For example, let's say we have this step:

        <batch:step id="createCsvStep">
         <batch:tasklet>
            <batch:chunk reader="jdbcCursorItemReader" processor="processor" writer="compositeWriter"
                         commit-interval="#{jobParameters['commit.interval']}" />
         </batch:tasklet>
        </batch:step>

        <bean id="compositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter" scope="step">
         <property name="delegates">
          <list>
            <ref bean="csvFileItemWriter1"/>
            <ref bean="csvFileItemWriter2"/>
          </list>
        </property>
       </bean>

Let's assume that first writer will write all the values, but in the same time, second writer will skip some of them. In order to achive this, we could extend our writer(for ex. FlatFileItemWriter), and override it write method like this:

    @Override
public void write(List<? extends T> items) throws Exception {
    // ...
    if (itemsPassesCheckingCondition) {
        super.write(items);
    }
}