MySQL Insert performance degrades on a large table

Question

I\'m working with a huge table which has 250+ million rows. The schema is simple.

CREATE TABLE MyTable (
        id BIGINT PRIMARY KEY AUTO_INCREMENT,
              


        

Answer 1

You didn't say whether this was a test system or production; I'm assuming it's production.

It is likely that you've got the table to a size where its indexes (or the whole lot) no longer fits in memory.

This means that InnoDB must read pages in during inserts (depending on the distribution of your new rows' index values). Reading pages (random reads) is really slow and needs to be avoided if possible.

Partitioning seems like the most obvious solution, but MySQL's partitioning may not fit your use-case.

You should certainly consider all possible options - get the table on to a test server in your lab to see how it behaves.

Your primary key looks to me as if it's possibly not required (you have another unique index), so eliminating that is one option.

Also consider the innodb plugin and compression, this will make your innodb_buffer_pool go further.

You really need to analyse your use-cases to decide whether you actually need to keep all this data, and whether partitioning is a sensible solution.

Making any changes on this application are likely to introduce new performance problems for your users, so you want to be really careful here. If you find a way to improve insert performance, it is possible that it will reduce search performance or performance of other operations. You will need to do a thorough performance test on production-grade hardware before releasing such a change.

Answer 2

I'll second @MarkR's comments about reducing the indexes. One other thing you should look at is increasing your innodb_log_file_size. It increases the crash recovery time, but should help. Be aware you need to remove the old files before you restart the server.

General InnoDB tuning tips: http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/

You should also be aware of LOAD DATA INFILE for doing inserts. It's much faster.

Answer 3

From my experience with Innodb it seems to hit a limit for write intensive systems even if you have a really optimized disk subsystem. I am surprised you managed to get it up to 100GB.

This is what twitter hit into a while ago and realized it needed to shard - see http://github.com/twitter/gizzard.

This all depends on your use cases but you could also move from mysql to cassandra as it performs really well for write intensive applications.(http://cassandra.apache.org)

Answer 4

Increase from innodb_log_file_size = 50M to innodb_log_file_size = 500M

And the innodb_flush_log_at_trx_commit should be 0 if you bear 1 sec data loss.

Answer 5

As MarkR commented above, insert performance gets worse when indexes can no longer fit in your buffer pool. InnoDB has a random IO reduction mechanism (called the insert buffer) which prevents some of this problem - but it will not work on your UNIQUE index. The index on (hashcode, active) has to be checked on each insert make sure no duplicate entries are inserted. If the hashcode does not 'follow' the primary key, this checking could be random IO.

Do you have the possibility to change the schema?

Your best bet is to:

(a) Make hashcode someone sequential, or sort by hashcode before bulk inserting (this by itself will help, since random reads will be reduced).

(b) Make (hashcode,active) the primary key - and insert data in sorted order. I am guessing your application probably reads by hashcode - and a primary key lookup is faster.

Answer 6

You didn't mention what your workload is like, but if there are not too many reads or you have enough main-memory, another option is to use a write-optimized backend for MySQL, instead of innodb. Tokutek claims 18x faster inserts and a much more flat performance curve as the dataset grows.

tokutek.com

http://tokutek.com/downloads/tokudb-performance-brief.pdf