I am trying to insert 100,000 rows in a MYSQL table under 5 seconds using Hibernate(JPA). I have tried every trick hibernate offers and still can not do better than 35 seconds.
You are using Spring for managing the transaction but break it by using thread
as the current session context. When using Spring to manage your transactions don't mess around with the hibernate.current_session_context_class
property. Remove it.
Don't use the DriverManagerDataSource
use a proper connection pool like HikariCP.
In your for loop you should flush
and clear
the EntityManager
at regular intervals, preferably the same as your batch size. If you don't a single persist takes longer and longer, because when you do that Hibernate checks the first level cache for dirty objects, the more objects the more time it takes. With 10 or 100 it is acceptable but checking 10000s of objects for each persist will take its toll.
-
@Service
@Transactional
public class ServiceImpl implements MyService{
@Autowired
private MyDao dao;
@PersistenceContext
private EntityManager em;
void foo(){
int count = 0;
for(MyObject d : listOfObjects_100000){
dao.persist(d);
count++;
if ( (count % 30) == 0) {
em.flush();
em.clear();
}
}
}
For a more in depth explanation see this blog and this blog.