Adding durability to in-memory data structures

Question

What are some of the popular techniques you can adopt to add durability to your in-memory data structures (ie) if the process crashes, you can preserve all previously ex

Answer 1

I've implemented the "Mrjb" technology in 2 companies' products, which is basically exactly what you've suggested in your question: a "Memory Resident Journal Backed" database, an in-memory data-structure where every change is logged to disk as it happens. And it works great for us!

http://www.edval.biz/memory-resident-programming-object-databases

I'd be happy to share our real-world experiences with using this in a production context. I love being able to replay an exact sequence of events or roll back to any point in time.

Answer 2

You might want to try an object prevalence engine. For .NET, you might want to try Bamboo.Prevalence, which is a port of a similar engine called Prevayler for Java.

Answer 3

The word you're looking for is "serialization".

Answer 4

Yes, you would want to serialize the data to some format - xml, binary, whatever. Depending on the programming languagem this may be built in for you. Java has ObjectStreams, .NET has XmlSerializer and also a BinaryFormatter.

Answer 5

Any answer to your question will entail doing something like what an ACID database system does. So I would say your best bet is to use an RDBMS to store your application state, updating whenever you have an (application) transaction that must not be lost.

Answer 6

You could come up with some way to serialize your structure, whether with XML, YAML, JSON, etc. Then you could either store that in the DB, or perhaps put one big try/catch around the main execution point to the program. Then if some uncaught exception happens, which will cause the program to crash, you could serialize your data, ans well as log any error messages, stack traces, etc.