I've hit the DB performance bottleneck, where now?

Question

I have some queries that are taking too long (300ms) now that the DB has grown to a few million records. Luckily for me the queries don\'t need to look at the majority of this d

Answer 1

I would start by trying to optimize the tables/indexes/queries before before taking any of the measures you listed. Have you dug into the poorly performing queries to the point where you are absolutely convinced you have reached the limit of your RDBMS's capabilities?

Edit: if you are indeed properly optimized, but still have problems, consider creating a Materialized View for the exact data you need. That may or may not be a good idea based on more factors than you have provided, but I would put it at the top of the list of things to consider.

Answer 2

Well, if you have optimised the database and queries, I'd say that rather than chop up the data, the next step is to look at:

a) the mysql configuration and make sure that it is making the most of the hardware

b) look at the hardware. You don't say what hardware you are using. You may find that replication is an option in your case if you can buy a two or three servers to divide up the reads from the database (writes have to be done to a central server, but reads can be read from any number of slaves).

Answer 3

I understand I'm now getting away from what the question was intended for so maybe I should make another one. My problem is that EXPLAIN is saying the query takes 10ms rather than 300ms which jprofiler is reporting.

Then your problem (and solution) must be in java, right?

Answer 4

Instead of creating a separate table for latest results, think about table partitioning. MySQL has this feature built in since version 5.1

---

Just to make it clear: I am not saying this is THE solution for your issues. Just one thing you can try

Answer 5

Considering a design change like this is not a good sign - I bet you still have plenty of performance to squeeze out using EXPLAIN, adjusting db variables and improving the indexes and queries. But you're probably past the point where "trying stuff" works very well. It's an opportunity to learn how to interpret the analyses and logs, and use what you learn for specific improvements to indexes and queries.

If your suggestion were a good one, you should be able to tell us why already. And note that this is a popular pessimization--

What is the most ridiculous pessimization you've seen?

Answer 6

Searching in the last 100,000 records should be terribly fast, you definitely have problems with the indexes. Use EXPLAIN and fix it.