Handling large databases

Question

I have been working in a web project(asp.net) for around six months. The final product is about to go live. The project uses SQL Server as the database. We have done performance

Answer 1

First off as many others have said a few million rows is not large. The current application I'm working on has several tables all with over a hundred million rows in which are all normalised.

We did suffer from some poor performance but this was caused by using the default table statistics settings. Inserting small numbers of records relative to the total size of the table, i.e. inserting a million records into a table containing 100+ million records wasn't causing an automatic update of the table stats and so we'd get poor query plans which manifested itself as serial queries being produced instead of parallel.

As to whether it's the right decision to denormalise, depends on your schema. Do you have to perform deep queries regularly i.e. loads of joins to get at data that you regularly need access to, if so then partial denormaisation might be a way forward.

BUT NOT BEFORE you've checked your indexing and table statistic strategies.
Check that you're using sensible, well structured queries and that your joins are well formed. Check your query plans that your queries are actually parsing the way you expect.

As others have said SQL Profiler/Database Engine Tuning Advisor do actually make a good job of it.

For me denormalisation is usually near the bottom of my list of things to do.

If you're still having problems then check your Server Software and Hardware setup.

• Are your database and log files on separate physical disks using separate controllers?
• Does it have enough memory?
• Is the log file set to autogrow? If so is the autogrow limit to low, i.e. is it growing to often.

Answer 2

At first we were using fully normailized database, but now we made it partially normailzed due to performance issues (to reduce joins).

As the old saying goes "normalize till it hurts, denormalise till it works".

It's fairly common in large, heavy-use dbs to see a degree of denormalisation to aid performance, so I wouldn't worry too much about it now, so long as your performance is still where you want it to be and your code to manage the "denormalised" fields doesn't become too onerous.

what are the possible solutions when data size becomes very large, as the no. of clients increase in future?

Not knowing too much about your application's domain, it's hard to say how you can future-proof it, but splitting out recently used and old data to separate tables is a fairly common approach in heavily-trafficked databases - if 95% of your users are querying their data from the last 30/45 days, having a "live_data" table containing, say, the last 60 day's worth of data and an "old_data" for the older stuff can help your performance.

A good idea would be to make sure you have extensive performance monitoring set up so that you can measure your db's performance as the data and load increases. If you find a noticeable drop in performance, it might be time to revisit your indexes!