How to speed up a data loading into InnoDB (LOAD DATA INFILE)?

问题

I want to speed up a data loading.

I use MySQL 5.5, InnoDB and have 1M rows of data (65Mb file). It takes 5 minutes.

What mysql settings and commands affect the speed of LOAD DATA INFILE for InnoDB?

Thank you.

回答1:

I can recommend these settings to improve load time:

innodb_doublewrite = 0
innodb_support_xa = 0
innodb_buffer_pool_size = (50-80% of system memory)
innodb_log_file_size = (a large number - 256M etc)
innodb_flush_log_at_trx_commit = 0

Other than settings, there are some things you can do yourself:

Create indexes after loading (this is a new optimization with 5.5 / InnoDB plugin).
Sort the data file before loading.
Split the data file, and load in parallel.

回答2:

Try removing indexes and triggers. You can re-create them after the load. Also look into using some of the high-load settings in my-huge.cnf instead of the defaults.

Some more innodb performance settings:

http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/

回答3:

This might not be exactly what you're looking for but is a trick I've used in the past

ALTER TABLE TABLE_NAME DISABLE KEYS;
LOAD DATA INFILE ... ;
ALTER TABLE TABLE_NAME ENABLE KEYS;

Hope it helps.

回答4:

Also make sure that binary logging disabled if possible.

回答5:

If you are in a hurry because you are replacing the contents of a live table, then do it this way instead:

CREATE TABLE new LIKE live;
LOAD DATA ... INTO new;
RENAME TABLE live TO old, new TO live;
DROP TABLE old;

The RENAME is 'instantaneous' and atomic, so you are 'never' down, regardless of table size.

(Hence you don't need to worry so much about speeding up the LOAD.)

回答6:

There seems to be a high correlation between insert time and file size. I recommend partitioning the files and inserting into partition tables in parallel.

r = 0.8936003754671448

Table size growth was not correlated with insert performance (bubble size = files size).

r = 0.004109728810459718

来源：https://stackoverflow.com/questions/5020998/how-to-speed-up-a-data-loading-into-innodb-load-data-infile

标签

mysql

import

InnoDB

load-data-infile