MySql - WAMP - Huge Table is very slow (20 million rows)

Question

So I posted this! yesterday and got a perfect answer, which required running this code first: ALTER TABLE mytable AUTO_INCREMENT=10000001;

I ran it several times, but r

Answer 1

It's going to take a long time because you've only got 2GB RAM and 6GB of data/indexes and it's going to force a ton of swapping in/out between RAM and disk. There's not much you can do about that, though.

You could try running this in batches.

Create a separate empty table with the auto_increment column included in it. Then insert your records a certain amount at a time (say, 1 state at a time). That might help it go faster since you should be able to handle those smaller datasets completely in memory instead of paging to disk.

You'll probably get a lot better responses for this if it's on dba.stackexchange.com also.

Answer 2

Some answers:

• 20 million rows is well within the capability of MySQL. I work on a database that has over 500 million rows in one of its tables. It can take hours to restructure a table, but ordinary queries aren't a problem as long as they're assisted by an index.

• Your laptop is pretty out of date and underpowered to use as a high-scale database server. It's going to take a long time to do a table restructure. The low amount of memory and typically slow laptop disk is probably constraining you. You're probably using default settings for MySQL too, which are designed to work on very old computers.

• I wouldn't recommend using TEXT data type for every column. There's no reason you need TEXT for most of those columns.

• Don't create an index on every column, especially if you insist on using TEXT data types. You can't even index a TEXT column unless you define a prefix index. In general, choose indexes to support specific queries.

You probably have many other questions based on the above, but there's too much to cover in a single StackOverflow post. You might want to take training or read a book if you're going to work with databases.
I recommend High Performance MySQL, 2nd Edition.

---

Re your followup questions:

For MySQL tuning, here's a good place to start: http://www.mysqlperformanceblog.com/2006/09/29/what-to-tune-in-mysql-server-after-installation/

Many ALTER TABLE operations cause a table restructure, which means basically lock the table, make a copy of the whole table with the changes applied, and then rename the new and old tables and drop the old table. If the table is very large, this can take a long time.

A TEXT data type can store up to 64KB, which is overkill for a phone number or a state. I would use CHAR(10) for a typical US phone number. I would use CHAR(2) for a US state. In general, use the most compact and thrifty data type that supports the range of data you need in a given column.

Answer 3

I believe the hardware is fine but you need to spare your resources a lot better.

Db structure optimization!

• Do not use TEXT!
• For phonenumbers use bigint unsigned. Any signs or alpha must be parsed and converted.
• For any other alpha-numeric column use eg varchar([32-256]).
• Zip-code is of course mediumint unsigned.
• Gender should be enum('Male','Female')
• Sales could be an int unsigned
• State should be enum('Alaska',...)
• Country should be enum('Albania',...)

When building a large index the fastest way is to create a new table and do INSERT INTO ... SELECT FROM ... rather then ALTER TABLE ....

Changing the State and Country fields to enum will drastically reduce you indexes size.