Has anyone published a detailed comparison between different in-memory RDBMSs? [closed]

There are quite a few independent and not-so-independent studies comparing traditional RDBMSs but I haven't managed to find any good material on in-memory databases. I am primarily interested in ones specialized for OLTP.

So far, I managed to find generic white papers on TimesTen and MySQL Cluster, but I have yet to see a head-to-head comparison. There are other alternatives (e.g. from IBM), but there's even less easily available material.

The information is scattered all over the web, but here's what I found out:

Introduction to database benchmarking

The first thing that you need to do is figure out a standard benchmark to use for comparing your in-memory database choices. A wiki page on Oracle gives an overview of the industry standard database benchmarks. If you can find results for the same industry standard benchmarks for TimesTen and MySql Cluster, you'll be able to compare the two. Finding a detailed comparison of all the in-memory databases is quite rare.

Summary: Since you mentioned OLTP, the industry standard benchmark that you're probably interested in is TPC-E which simulates the OLTP workload of a brokerage firm. Other industry standard benchmarks measure decision support and e-commerce database transactions.

Open source database benchmark kits

The Open Source Development Labs Database Test Suite is the most comprehensive open source implementation of those industry standard benchmarks I mentioned earlier. They have four main suite of tests numbered DBT1, DBT2, DBT3, DBT4. They haven't implemented the TPC-E benchmark, but the Database Test 2 (DBT2) is what you are looking for since it simulates OLTP transactions. Everyone simply refers to running the DBT2 benchmark instead of spelling it out as OSDLB DBT2.

Results

MySql

MySql Cluster Website

MySql published benchmarks

You'll need to register to get the white paper emailed to you, but here is the summary that I took from their site:

Ultimately, MySQL Cluster was able to reach 100,000 transactions per minute in an 8-node configuration. This showed impressive scaling improvements when compared to a 2-node cluster which was able to obtain 26,000 transactions per minute. These performance improvements were realized through the use of new Multi Core Intel Xeon servers based on the Intel Core T micro architecture.

Oracle

Oracle TimesTen Website

Unfortunately the DBT2 does not support TimesTen. However I found a whitepaper that gives detailed benchmarking information for TimesTen in an OLTP workload.

IBM

IBM SolidDB Website

Other Resources

In Memory DBMS section of a blog that focuses on databases.
Transaction Processing Performance Council website - You can search their database for benchmarks from various hardware vendors using various DBMSes. I was unable to find any in memory DBMS benchmarks however.

tmcallaghan

VoltDB is an in-memory highly-scalable OLTP database that provides SQL/ACID and runs on commodity hardware in a shared-nothing cluster. It is open-source and exists in both a free/community (AGPLv3) version as well as a commercially supported version.

There is a post in the VoltDB forums discussing some of the other in-memory OLTP RDBMS solutions at http://community.voltdb.com/node/95. Feel free to review and add to that posting.

Sebastian Czechowski at BigDataMatters has a comparison of four different IMDBs, suggesting selects tend to be 5X faster, and inserts/deletes tend to be 2x-4x faster.

http://bigdatamatters.com/bigdatamatters/2009/12/oracle_ibm_database_comparison.html

This wiki article gives a comparison of most of the well known RDBMS's: Comparison of relational database management systems

No comparison, but lists the in-memory RDBMS's: In-memory database

Also, it is going to depend on your intended use to some extent. What are your constraints and minimum performance requirements?

I think your best option is to obtain trial (or free) versions and set up your own benchmarks.

Almost every DBMS will do the same thing but differently, therefore you should look for the fastest DBMS you can find, if what you need is speed then you should use TimesTen wich is accepted as one of the fastest solutions, the thing with open-source DBMS is that, some of the more advanced features (such as replication) are nowhere near what you'd find on commercial alternatives. Quite simply, most users don't need replication at the levels that Oracle, DB2, or MS-SQL offer; therefore, PostgreSQL and MySQL developers don't get feel the need to improve it.

This answer was obtained after looking for the fastest between RDBMS mentioned here and some already known by me, I went 2 by 2 looking for the best and then looking for the best between the winner and the other, and the result was that TimesTen is the fastest between: