Spring Batch - Is it possible to have a dynamic column in FlatFileReader?

Question

I\'m dealing with many CSVs files that don\'t have a fixed header/column, saying that I can get file1.csv with 10 column and file2.csv with 50 column.

I can\'t know

Answer 1

I arrived to this old post with the very same question. Finally I managed to build a dynamic column FlatFileItemReader with the help of the skippedLinesCallback so I leave it here:

@Bean
public FlatFileItemReader<Person> reader() {

    DefaultLineMapper<Person> lineMapper = new DefaultLineMapper<>();
    DelimitedLineTokenizer delimitedLineTokenizer = new DelimitedLineTokenizer();
    lineMapper.setLineTokenizer(delimitedLineTokenizer);
    lineMapper.setFieldSetMapper(new BeanWrapperFieldSetMapper<>() {
        {
            setTargetType(Person.class);
        }
    });

    return new FlatFileItemReaderBuilder<Person>()
            .name("personItemReader")
            .resource(new FileSystemResource(inputFile))
            .linesToSkip(1)
            .skippedLinesCallback(line -> delimitedLineTokenizer.setNames(line.split(",")))
            .lineMapper(lineMapper)
            .build();
}

In the callback method you update the names of the tokenizer from the header line. You could also add some validation logic here. With this solution there is no need to write your own LineTokenizer implementation.

Answer 2

Create your own LineTokenizer implementation. The DelimitedLineTokenizer expects a predefined number of columns. If you create your own, you can be as dynamic as you want. You can read more about the LineTokenizer in the documentation here: http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/item/file/transform/LineTokenizer.html