Very large tables in SQL Server

Question

We have a very large table (> 77M records and growing) runing on SQL Server 2005 64bit Standard edition and we are seeing some performance issues. There are up to a hundred

Answer 1

In and of itself, 77M records is not a lot for SQL Server. How are you loading the 100,000 records? is that a batch load each day? or thru some sort of OLTP application? and is that the performance issue you are having, i.e adding the data? or is it the querying that giving you the most problems?

If you are adding 100K records at a time, and the records being added are forcing the cluster-index to re-org your table, that will kill your performance quickly. More details on the table structure, indexes and type of data inserted will help.

Also, the amount of ram and the speed of your disks will make a big difference, what are you running on?

Answer 2

maybe these are minor nits, but.... (1) relational databases don't have FIELDS... they have COLUMNS. (2) IDENTITY columns usually mean the data isn't normalized (or the designer was lazy). Some combination of columns MUST be unique (and those columns make up the primary key) (3) indexing on datetime columns is usually a bad idea; CLUSTERING on datetime columns is also usually a bad idea, especially an ever-increasing datetime column, as all the inserts are contending for the same physical space on disk. Clustering on datetime columns in a read-only table where that column is part of range restrictions is often a good idea (see how the ideas conflict? who said db design wasn't an art?!)

Answer 3

What type of disks do you have?

You might monitor some disk counters to see if requests are queuing.

You might move this table to another drive by putting it in another filegroup. You can also to the same with the indexes.

Answer 4

http://msdn.microsoft.com/en-us/library/ms143432.aspx

You've got some room to grow.

As far as performance issues, that's a whole other question. Caching, sharding, normalizing, indexing, query tuning, app code tuning, and so on.

Answer 5

The first thing I'd look at is indexing. If you use the execution plan generator in Management Studio, you want to see index seeks or clustered index seeks. If you see scans, particularly table scans, you should look at indexing the columns you generally search on to see if that improves your performance.

You should certainly not need to move to Enterprise edition for this.

Answer 6

Do you really need to have access to all 77 million records in a single table?

For example, if you only need access to the last X months worth of data, then you could consider creating an archiving strategy. This could be used to relocate data to an archive table in order to reduce the volume of data and subsequently, query time on your 'hot' table.

This approach could be implemented in the standard edition.

If you do upgrade to the Enterprise edition you can make use of table partitioning. Again depending on your data structure this can offer significant performance improvements. Partitioning can also be used to implement the strategy previously mentioned but with less administrative overhead.

Here is an excellent White paper on table partitioning in SQL Server 2005

http://msdn.microsoft.com/en-us/library/ms345146.aspx

I hope what I have detailed is clear and understandable. Please do feel to contact me directly if you require further assistance.

Cheers,