What is the best way to achieve speedy inserts of large amounts of data in MySQL?

Question

I have written a program in C to parse large XML files and then create files with insert statements. Some other process would ingest the files into a MySQL database. This

Answer 1

I'd do at least these things according to this link:

1. Move the files there and connect over the unix socket
2. Generate, instead of the INSERTS, a LOAD DATA INFILE file
3. Disabling indexes during the loading

Answer 2

1. Make sure you use a transaction.

Transactions eliminate the

INSERT, SYNC-TO-DISK

repetition phase and instead all the disk IO is performed when you COMMIT the transaction.

2. Make sure to utilize connection compression

Raw text + GZip compressed stream ~= as much as 90% bandwidth saving in some cases.

3. Utilise the parallel insert notation where possible

INSERT INTO TableName(Col1,Col2) VALUES (1,1),(1,2),(1,3) 

( Less text to send, shorter action )

Answer 3

MySQL with the standard table formats is wonderfully fast as long as it's a write-only table; so the first question is whether you are going to be updating or deleting. If not, don't go with innosys - there's no need for locking if you are just appending. You can truncate or rename the output file periodically to deal with table size.

Answer 4

Really depends on the engine. If you're using InnoDB, do use transactions (you can't avoid them - but if you use autocommit, each batch is implicitly in its own txn), but make sure they're neither too big or too small.

If you're using MyISAM, transactions are meaningless. You may achieve better insert speed by disabling and enabling indexes, but that is only good on an empty table.

If you start with an empty table, that's generally best.

LOAD DATA is a winner either way.

Answer 5

• Use the mysqlimport tool or the LOAD DATA INFILE command.
• Temporarily disable indices that you don't need for data integrity

Answer 6

If you can't use LOAD DATA INFILE like others have suggested, use prepared queries for inserts.