looking for light-weight data persistence solution in perl

Question

In my app I need to store some simple data both in memroy and in disk. A real database will be overkill in my case, so I need lighter one to handle the simple data persistence r

Answer 1

Storable lets you serialize any Perl data structure and read it back in. For in-memory storage, just use IO::Scalar to store into a string, that way you only need to write the code once and for writing to disk you just pass in another I/O handle.

Answer 2

You have several options:

• Storable is a core module and is very efficient. It has some problems with portability, for example someone using an older version of Storable may not be able to read your data. Also, the endianness of the system creating and retrieving that data is important. The network order stoarge options help reduce the portability issues. You can store an arbitrary nested data structure to a file or string and restore it. Storable is supported only by Perl.

• YAML is a text based format that works like storable--you can store and restore arbitrary structures to/from YAML files. YAML is nice because there are YAML libraries for several languages. It's not quite as speedy or space efficient as Storable.

• JSON is a popular data exchange format with support in many languages. It is very much like YAML in both its strengths and weaknesses.

• DBD::SQLite is a database driver for the DBI that allows you to keep a whole relational database in a single file. It is powerful and allows you work with many of the persistence tools that are aimed at other databases like MySQL and Postgres.

• DBM::Deep is a convenient and powerful perl only module that allows efficient retrieval and modification of small parts of a large persistent data structures. Almost as easy to use as Storable, but far more efficient when dealing with only small portions of a large data structure.

Update: I realized that I should mention that I have used all of these modules and depending on your particular needs, any of them could be "the right choice".

Answer 3

Look into Tie::File and submodules like Tie::File::AsHash, or Tie::Handle::CSV. All available on CPAN, fast and easy to use.

Answer 4

You might want to try Tie::Storable. Then it's as simple as addressing a hash.

If you're not looking to store a ton of data and you're OK loading everything all at once at program startup, it might be the way to go.

If you're looking for something more sophisticated but still light weight, a lot of people (including myself) swear by SQLite.

Answer 5

Do you want your data to be transparently persisted, i.e. you won't have to worry about doing a commit()-type operation after every write? I just asked a very similar question: Simple, modern, robust, transparent persistence of data strutures for Perl, and listed all the solutions I found.

If you do want transparent persistence (autocommit), then DBM::Deep may be easier to use than Storable. Here is example code that works out of the box:

use DBM::Deep;

tie my %db, 'DBM::Deep', 'file.db';

if ( exists $db{foo}->{bar} ) {
    print $db{foo}->{bar}, "\n"
} else {
    $db{foo}->{bar} = 'baz';
}    

Answer 6

If I had to do this I would probably go with DBI and DBD::SQLite, since it does not involve reading all the data into memory, but I'd just like to mention a few other ways, because "there's more than one way to do it":

The old way to do this was with DB_file and its cousins. It still works with modern versions of Perl. The drawback is that it are only useful for storing a one-dimensional hash (a hash which doesn't have any references in it). The advantage is that you can find nice books about it which don't cost very much money, and also online articles, and also I believe it doesn't involve reading the whole file into memory.

Another method is to print the contents of Data::Dumper to a file to store, and eval the contents of the file to read the data.

Yet another thing which hasn't been mentioned is KiokuDB, which looks like the cutting-edge Moose-based module, if you want to be trendy.