When should I consider using a in memory database and what are the issue to look out for?

Question

I was just think that now it is common to have enough RAM on your database server to cache your complete database why are the specialist in memory database (e

Answer 1

Most probably there are just no mature products of memory databases which could be used as a full replacement for a classic database.

Relational database are a very old concept. Although there were many approaches to move forward and develop new technologies, eg. object oriented databases, the relational databases didn't really change their concepts. Don't expect things to change too fast, since databases didn't change much in the last ten or fifteen years or even longer.

I think, development of technologies is not as fast as one might believe. It takes decades for new concepts to be matured and established. First of all in database technologies, where maturity is much more important then anything else.

In ten or twenty years, databases are probably not the same anymore as they are today. If in-memory databases are the future - nobody can tell this today - they just need some more time to develop.

Answer 2

Nobody really answered the question "When should I consider using a in memory database and what are the issue to look out for?" so I'll give it a go.

You should consider an in-memory database if: 1. The target system has data to manage, but no persistent media 2. The performance requirement simply cannot be met with a persistent database

For #1, think of the TV Guide in your set-top box (STB). Low-end STB (i.e. those with no DVR capability) have no persistent storage, and need no persistent storage. But the database for a 400-channel, 14-day TV Guide is non-trivial. There's a performance requirement here, too, because data arrives from the transponder carousel at a high speed and it's a case of 'capture it or wait until the carousel comes around again'. But there's no need for persistence. We've all seen this; when you lose power at your home, when it comes back on the TV Guide says "will be available shortly" because it's provisioning itself from the transponder or cable head-end. Network routers share the same characteristics: no persistent storage, need to be fast, and the database can be provisioned from an external source (peer routers on the network, in this case, to repopulate the routing table).

There are endless examples of #2: Real-time targetting in military systems, high-frequency trading systems, and more.

Regarding the second part of the question, "issue to watch out for": There are many.

Make sure you're evaluating a true in-memory database if you need the performance that only an in-memory database can deliver. Caching a persistent database is not the same. Throwing a persistent database in a RAM-drive is not the same. Using an in-memory database that inherently does transaction logging (like TimesTen) is not the same (even if you log to /dev/null).

Make sure you're evaluating a database system, and not merely a cache (e.g. memcache). A database system will have support for transactions with the ACID properties, multiple indexing options, support concurrent access, and more.

About ACID: in-memory database systems do not lack the 'D' (durability). It simply has to be taken in context. Transactions in a persistent database are durable only so long as the media it's stored on is durable. The same thing is true for in-memory databases. In either case, if you care about durability, you better have a backup.

Answer 3

Various Portable versions of SQL, that will work with same efficiency, designed for mobile devices mainly.

SQLite

SQL Server Compact Edition

These are just big players other options may be there, but big players handle minimum requirements with release of it.. :)

and in memory database, you have continuously back up the data if fluctuation or powercut arises you may loss the whole bunch. as in other that will be handled as its in secondary memory(HDD) and the chances of loss will be 10% compare to memory DB.

I hope this may help :)

Answer 4

This is also an option: http://www.memsql.com/

I have not used it personally, but it's supposed to be along the lines of a drop-in replacement for MySQL in-memory.

Answer 5

The most typical use-case for a database is persistence, which makes most in-memory databases unsuitable. One popular reason to use an in-memory database is for testing purposes. But this requires you use either a database that can be set up both as in-memory and something else.

Popular choices in this area seems to be RavenDB for .Net developers and OrientDB for Java developers. Because both can function as in-memory databases, and "something else" depending on configuration, so you can use one or the other depending on your configuration (app.config in .Net, Maven or Ant settings in Java).

Answer 6

Data processing needs are getting more complex and the product ecosystem is evolving to meet these new needs. Disk-based RDBMS, in-memory cache, and in-memory databases are used to satisfy different needs. You should go with what suits your need -

Traditional RDBMS: Your MySQL cluster is fast enough, easy enough to maintain, and you like having the reliability of ACID-compliance.

In-memory distributed cahce: Your application needs to do fast reads and writes without worrying too much about consistency or complex transactions.

In-memory RDBMS:

1. (Speed): Your application needs to process data/requests faster than your disk-based database can.
2. (Complexity): You need to make complex transactional reads and writes with joins and aggregations and like to use the power of SQL.
3. (Scalability): You need to scale your database horizontally without downtime.
4. (Maintainability): You need the database to provide high availability, replication, load-balancing and disaster recovery without adding to your maintenance chores.
5. (Caveat): Your data should fit in memory (typically in terabytes).