How do I design a table which will store very large data?

Question

I need to design a table in Oracle, which will store 2-5 TB of data in a day. It can grow to 200TB, and records will purged when it crosses 200 TB.

Is it a feasible choice to keep it in OLTP, or do I need to shift it to data warehouse DB? Please advice considerations I should keep in mind when designing the schema of this table, or the database.

Also, please advice if it is a SQL server, as I can use either database.

Answer 1

That size puts you in the VLDB territory (very large databases). Things are fundamentally different at that altitude.

Your question cannot be answered without the full requirements of the responsibilities of your application. You need to design for performance with respect to what your application should DO with the data.

My advice is to get someone onboard who have previous experience, or you are close to 100% guaranteed to fail.

If you go with Oracle, it provides several types of partitioning that you will want to use very carefully. You need partitions for administrative purposes (moving data, building indexes, recovering data) as well as for query performance:

• Range partitioning, for example by a date range
• List partitioning, for storing slices of data say country-wise ('SE', 'US', 'GB')
• Hash partitioning. Stores your data in one of the partitions based on a hash function
• Or any combination of the above

Also, you need someone who knows how to build and configure a monster machine with truly awesome I/O throughput. You need more than 1GB/s, which isn't very cheap when you also need to store 200 TB. Actually, if those 200 TB are table data only, you will need to double or triple that to be able to create indexes, aggregate tables, backups etc.

Sorry I couldn't give you a solution ready for use, but I wanted to make sure you understand that you are not just building an above average sized database. It is massive!

Answer 2

I would first look at your disk subsystem and make sure that it can handle the 2-5 TB load that you will do every day. Both Oracle and SQL Server have partitioning built in. In SQL Server you can also use compression to minimize storage on disk(not sure about Oracle)

Will this be mostly used for OLAP or OLTP?