问题
Summary: Inserting 100k rows at a time with 1k batches (yes it's definitely being batched) and time to insert is increasing for each batch. Trying to understand why.
Original Question: JProfiler shows a lot of time spent reading an InputStream in ReadAheadInputStream.fill.
Update: Per comment below from EJP "The time is being spent blocked in read(), while the database server is executing the statement and preparing the response"
What could cause so much time to be spent here?
Details:
- Inserting rows into a single table in 1k batches.
- Batches are inside a transaction which triggers flush of Hibernate session.
- Hibernate and MySQL connection batch properties are enabled and profiling confirms SQL INSERT is indeed batched as expected.
- As the table grows, the time to insert grows (ie. first 100k inserted in 25 seconds, second 100k in 30, 35, 40, etc).
We have another app (which used JDBC Template) which executes the same INSERT on the same table but does NOT slow down as the table grows. Therefore while db tuning may improve performance, since another app basically does the same insert without performance issues, sub-optimal DB config does not explain the slow down.
Dev Stack:
- Java 8
- Hibernate 4.3.6
- Spring Data JPA 1.7.2
- MySQL 5.6.20
- MySQL driver 5.1.33
- HikariCP 2.3.2
I understand I'm not providing a code sample. I'm basically looking for ideas and a better understanding of what might cause so much time spent in ReadAheadInputStream.fill and below in the call stack.
回答1:
In the end, we were unable to identify the root cause and fix the degradation issue for batch inserts using JPA/Hibernate. Therefore for large batch inserts we have reverted to using JDBC Template. We no longer see any performance degradation.
来源：https://stackoverflow.com/questions/31152076/why-are-the-insert-batch-times-increasing