improve speed of mysql import

Question

I have large database of 22GB. I used to take backup with mysqldump command in a gzip format.

When i extract the gz file it produces the

Answer 1

The method descirbed in [Vinbot's answer above][1] using LOAD DATA INFILE is how I bring in about 1 Gb every day for an analysis process on my local desktop (I don't have DBA or CREATE TABLErights on the server, but I do on my local mySQL).

A new feature introduced in mySQL 8.0.17, the [mySQL Parallel Table Import Utility][2], takes it to the next level.

An import of CSV tables that formerly took about 15 minutes (approx 1 Gb) now takes 5:30, on an Intel Core I7-6820HQ with a SATA SSD. When I added an nVME M.2 1Tb WD Black drive (bought for an old desktop but proved incompatible) and moved the mySQL installation to that drive, the time dropped to 4 min 15 sec.

I define most of my indexes in table definitions prior to running the utility. THe loads are even faster without indexing, but post-load indexing ends up taking more total time. This makes sense, as the multi-core feature of the Parallel Loader extends to index creation.

I also ALTER INSTANCE DISABLE INNODB REDO_LOG (introduced 8.0.21) in the parallel loader utility script. Heed the warning not to leave this off after finished with the bulk load. I did not reenable and ended up with corrupted instance (not just tables, but the whole instance). I keep double-write buffering off, always.

The CPU monitor shows the utility fully utilizes all 8 cores.

Once done with the parallel loader, it's back to single-threaded mySQL (for my linear set of analysis tasks, not multi-user). The new nVME cuts times by 10% or so. The utility saves me several minutes, every day.

The utility allows you to manage buffer sizes and number of threads. I match the number of physical cores in my CPU (8) and that seems optimal. (I originally came to this thread looking for optimization tips on configuring the parallel loader). [1]: https://stackoverflow.com/a/29922299/5839677 [2]: https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-utilities-parallel-table.html