Need to insert 100000 rows in mysql using hibernate in under 5 seconds

前端 未结 4 473
我寻月下人不归
我寻月下人不归 2021-01-30 23:38

I am trying to insert 100,000 rows in a MYSQL table under 5 seconds using Hibernate(JPA). I have tried every trick hibernate offers and still can not do better than 35 seconds.

4条回答
  •  抹茶落季
    2021-01-31 00:02

    1. You are using Spring for managing the transaction but break it by using thread as the current session context. When using Spring to manage your transactions don't mess around with the hibernate.current_session_context_class property. Remove it.

    2. Don't use the DriverManagerDataSource use a proper connection pool like HikariCP.

    3. In your for loop you should flush and clear the EntityManager at regular intervals, preferably the same as your batch size. If you don't a single persist takes longer and longer, because when you do that Hibernate checks the first level cache for dirty objects, the more objects the more time it takes. With 10 or 100 it is acceptable but checking 10000s of objects for each persist will take its toll.

    -

    @Service
    @Transactional
    public class ServiceImpl implements MyService{
    
        @Autowired
        private MyDao dao;
    
        @PersistenceContext
        private EntityManager em;
    
    
        void foo(){
            int count = 0;
            for(MyObject d : listOfObjects_100000){
                dao.persist(d);
                count++;
                if ( (count % 30) == 0) {
                   em.flush();
                   em.clear();
                }    
            }
        }
    

    For a more in depth explanation see this blog and this blog.

提交回复
热议问题