Database insert performance

Question

We are planning to implement a system for logging a high frequency of market ticks into a DB for further analysis. To simply get a little what kind of storage performance we can

Answer 1

I would consider checking MySQL 5.5 release candidate as well. Oracle guys made significant improvements in this version, specially for windows release. Up to 1,500 percent performance gains for Read/Write operations, and up to 500 percent gain for Read Only. You can refer to this link for more info:

http://www.mysql.com/news-and-events/generate-article.php?id=2010_04

Answer 2

There are many ways to optimize performance and different databases handle data very different as well. SQL Server for example is protecting your data, it has to be sure the data is valid and on disk before it lets you know the insert has been succesfull. MySQL nor MongoDB is doing that, so they can be faster. So, what are you looking for? A RDBMS or some storage where you can afford it to loose some data?

Answer 3

BerkeleyDB might be worth a look if your data can be represented as key/value pairs (as if in a PERL hash or similar data structure). It's fast, multiclient, and transaction-safe, even if it is not the latest wizbang thing.

Answer 4

Did you test with several application instances connected the the Database Server and inserting data at the same time or just one application?

I think you should be testing with multiple instances especially for bulk insertion and see what configuration works for you. Different transaction isolation modes can greatly affect the performance for concurrent access (especially write access). SQL Server for example, I found that lower isolation mode than ReadCommitted should be used for highly concurrent environment or you will find lot of cases of timeout. This should of course be used when the risk of dirty read is not a concern (which fit your case judging from your description).

PS: Forgive me if I'm stating the obvious here.

Answer 5

How do these compare to simply logging to a flat file in the file system? If querying is done later, I'm not sure why you are bringing the data into a relational database at this time. Is there a need for transactions or multiple access to the database at all during this recording stage?

Answer 6

The purpose of this testing is simply to get a little indication of what kind of "raw performance" can be expected of the systems in the bottom. When actually implementing a solution we would of course do buffering, bulk inserts etc.

You could at least share the details of your tests. Omitting such crucial information as what MySQL engine you try is unpardonable. And the "raw performance" of a non-batched insert on a buffer-pooled based DB (like SQL Server or InnoDB) is non-sense, is like measuring the "raw performance" of a Ferrari in first-gear and then publishing that "it only goes to 50mph".

But anyway, if you want a highly scalable write optimized DB, look at Cassandra from Apache Incubation. The rumor mill says Twitter will adopt it soon.