Mybatis return large result with xml configuration in spring

Question

I need to dump data from a table in oracle to elasticsearch(100 Million record), My memory limit of JVM is 256M, I use the following code and config to get the data from oracle

Answer 1

There are two options:

1. use ResultHandler
2. since 3.4.1 use Cursor

ResultHandler

That is how you can use custom ResultHandler:

PersonMapper.xml

<mapper namespace="com.fudy.mapper.PersonMapper">
  <resultMap type="com.fudy.domain.Person" id="PersonAlias">
    <id column="ID" property="id" />
    <result column="NAME" property="name" />
  </resultMap>
  <select id="selectAllPerson" resultMap="PersonAlias">
     SELECT * FROM person
  </select>
</mapper>

PersonMapper.java

public interface PersonMapper {
     void selectAllPersons(ResultHandler handler);
}

MyService.java

class PersonResultHandler implements ResultHandler {
    @Override
    public void handleResult(ResultContext context) { 
        Person person = (Person)context.getResultObject(); 
        // process person here
    }
};
PersonResultHandler handler = new PersonResultHandler();
PersonMapper personMapper = ...;
personMapper.selectAllPersons(handler);

Cursor

Starting from mybatis 3.4.1 you can return Cursor which is Iterable and can be used like this (under condition that result is ordered, see above Cursor API java doc for details):

PersonMapper.java

public interface PersonMapper {
     Cursor<Person> selectAllPersons();
}

MyService.java

PersonMapper personMapper = ...;
try (Cursor<Person> persons = personMapper.selectAllPersons()) {
   for (Person person : persons) {
      // process one person
   }
}

Answer 2

I can see that Roman Konoval gave you a good answer for your question but I what to add something.

I had a similar issue where I wanted to iterate through a table and not to get it as a one huge list. I chose a ResultHandler solution but I found some troubles.

Even if you use a ResultHandler it might not work as you expect. I might seem to work properly until you use a very large table.

I found that two things are important :

• a place where you put your transaction
• setting a fetchSize parameter

Until I set it properly MyBatis consumes memory and get whole table before it runs ResultHandler. When I set it MyBatis doesn't consume memory and runs ResultHandler immediately.

I prepared four cases :

1. a transaction is not set in a proper place.
2. a transaction is set but not a fetchSize parameter.
3. a fetchSize parameter is set but no transaction.
4. a transaction and a fetchSize parameter is set.

The first three cases lead to "java.lang.OutOfMemoryError: Java heap space" on my computer. The last case runs ResultHandler immediately and it doesn't consume memory.

My examples use a postgres database and generate some pseudo-table but of course it might be any large table. A table in my examples has 1_000_000 rows.

MyEntity.java:

public class MyEntity {
    private Long id;

    private String text;

    public void setId(Long id) {
        this.id = id;
    }

    public void setText(String text) {
        this.text = text;
    }

    @Override
    public String toString() {
        return "MyEntity{" + "id=" + id + ", text='" + text + '\'' + '}';
    }
}

MyMapper.java:

@Mapper
@Component
public interface MyMapper {
    void selectAll1(ResultHandler handler);

    void selectAll2(ResultHandler handler);
}

MyResultHandler.java:

class MyResultHandler implements ResultHandler {
    @Override
    public void handleResult(ResultContext context) {
        System.out.println((MyEntity) context.getResultObject());
    }
}

resources/application.properties :

spring.datasource.url=jdbc:postgresql://localhost/temp1
spring.datasource.username=user1
spring.datasource.password=user1
mybatis.config-location=classpath:mybatis-config.xml

resources/mybatis-config.xml :

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
        PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
    <mappers>
        <mapper resource="MyMapper.xml"/>
    </mappers>
</configuration>

resources/MyMapper.xml :

<!DOCTYPE mapper
        PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.example.mybatishugeselect.MyMapper">

    <select id="selectAll1" resultType="com.example.mybatishugeselect.MyEntity">
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    </select>

    <select id="selectAll2" fetchSize="1000" resultType="com.example.mybatishugeselect.MyEntity">
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    </select>

</mapper>

MybatisHugeSelectApplicationTests.java :

@SpringBootTest
class MybatisHugeSelectApplicationTests {

    /* no transaction , no fetchSize parameter */
    @Test
    void case1(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no fetchSize parameter */
    @Test
    @Transactional
    void case2(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no transaction */
    @Test
    void case3(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }

    /* good */
    @Test
    @Transactional
    void case4(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }
}

Only the last one ( case4 ) works properly.