Mybatis return large result with xml configuration in spring

Question

I need to dump data from a table in oracle to elasticsearch(100 Million record), My memory limit of JVM is 256M, I use the following code and config to get the data from oracle

Answer 1

I can see that Roman Konoval gave you a good answer for your question but I what to add something.

I had a similar issue where I wanted to iterate through a table and not to get it as a one huge list. I chose a ResultHandler solution but I found some troubles.

Even if you use a ResultHandler it might not work as you expect. I might seem to work properly until you use a very large table.

I found that two things are important :

• a place where you put your transaction
• setting a fetchSize parameter

Until I set it properly MyBatis consumes memory and get whole table before it runs ResultHandler. When I set it MyBatis doesn't consume memory and runs ResultHandler immediately.

I prepared four cases :

1. a transaction is not set in a proper place.
2. a transaction is set but not a fetchSize parameter.
3. a fetchSize parameter is set but no transaction.
4. a transaction and a fetchSize parameter is set.

The first three cases lead to "java.lang.OutOfMemoryError: Java heap space" on my computer. The last case runs ResultHandler immediately and it doesn't consume memory.

My examples use a postgres database and generate some pseudo-table but of course it might be any large table. A table in my examples has 1_000_000 rows.

MyEntity.java:

public class MyEntity {
    private Long id;

    private String text;

    public void setId(Long id) {
        this.id = id;
    }

    public void setText(String text) {
        this.text = text;
    }

    @Override
    public String toString() {
        return "MyEntity{" + "id=" + id + ", text='" + text + '\'' + '}';
    }
}

MyMapper.java:

@Mapper
@Component
public interface MyMapper {
    void selectAll1(ResultHandler handler);

    void selectAll2(ResultHandler handler);
}

MyResultHandler.java:

class MyResultHandler implements ResultHandler {
    @Override
    public void handleResult(ResultContext context) {
        System.out.println((MyEntity) context.getResultObject());
    }
}

resources/application.properties :

spring.datasource.url=jdbc:postgresql://localhost/temp1
spring.datasource.username=user1
spring.datasource.password=user1
mybatis.config-location=classpath:mybatis-config.xml

resources/mybatis-config.xml :

resources/MyMapper.xml :

    
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    

    
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    

MybatisHugeSelectApplicationTests.java :

@SpringBootTest
class MybatisHugeSelectApplicationTests {

    /* no transaction , no fetchSize parameter */
    @Test
    void case1(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no fetchSize parameter */
    @Test
    @Transactional
    void case2(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no transaction */
    @Test
    void case3(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }

    /* good */
    @Test
    @Transactional
    void case4(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }
}

Only the last one ( case4 ) works properly.