Mysql improve SELECT speed

Question

I\'m currently trying to improve the speed of SELECTS for a MySQL table and would appreciate any suggestions on ways to improve it.

We have over 300 million records

Answer 1

Your query is asking for a few things - and with that high # of rows, the look of the data can change what the best approach is.

   SELECT date, value 
   FROM table 
   WHERE tag = "a" 
     AND date BETWEEN 'x' and 'y' 
   ORDER BY date

There are a few things that can slow down this select query.

1. A very large result set that has to be sorted (order by).
2. A very large result set. If tag and date are in the index (and let's assume that's as good as it gets) every result row will have to leave the index to lookup the value field. Think of this like needing the first sentence of each chapter of a book. If you only needed to know the chapter names, easy - you can get it from the table of contents, but since you need the first sentence you have to go to the actual chapter. In certain cases, the optimizer may choose just to flip through the entire book (table scan in query plan lingo) to get those first sentences.
3. Filtering by the wrong where clause first. If the index is in the order tag, date... then tag should (for a majority of your queries) be the more stringent of the two columns. So basically, unless you have more tags than dates (or maybe than dates in a typical date range), then dates should be the first of the two columns in your index.

A couple of recommendations:

1. Consider if it's possible to truncate some of that data if it's too old to care about most of the time.
2. Try playing with your current index - i.e. change the order of the items in it.
3. Do away with your current index and replace it with a covering index (has all 3 fields in it)
4. Run some EXPLAIN's and make sure it's using your index at all.
5. Switch to some other data store (mongo db?) or otherwise ensure this monster table is kept as much in memory as possible.

Answer 2

I think that the value column is at the bottom of your performance issues. It is not part of the index so we will have table access. Further I think that the ORDER BY is unlikely to impact the performance so severely since it is part of your index and should be ordered.

I will argument my suspicions for the value column by the fact that the partitioning does not really reduce the execution time of the query. May you execute the query without value and further give us some results as well as the EXPLAIN? Do you really need it for each row and what kind of column is it?

Cheers!