Combining noSQL and ORM in an MVC framework for a real-case application

Question

I\'ve been trying for some time to put some \'cool\' things I\'ve been reading about noSQL (couchDB, mongoDB, Redis...) in the past years into practical use.

I\'m quite

Answer 1

This is what I get for trolling stackoverflow. Once in a great while an outstanding question gets asked and I am compelled to offer my 2 cents (at the risk of my own project timelines).

I just finished up a project where I had to de-couple an ORM from the model so I could implement a NoSQL solution, and found it not that difficult, although it was rough at times trying to figure out the best approach. So without getting too specific about my implementation, I will touch on what I had to do to make it work, as it may offer some enlightenment when you travel down the same path.

My setup:

• Framework - Symfony 1.4
• ORM - Doctrine 1.4
• NoSQL - My own proprietary solution

Goal:

• Store image paths within xml files vs the database
• Store html description paths within xml files vs the database

I didn't want to store images as blobs within the persistent store (database), and I didn't want to store image paths in the database, as I didn't want to pay the overhead of creating a database connection and querying for the path. So I decided to store the path information within a NoSQL persistent store (filesystem).

And ditto for html descriptions, I didn't want to create a text column on my table and store what could potentially be hundreds of lines of html within the database, and the same reasons as above.

All my NoSQL files relate to an object (refrigerator for example). These files contain paths to their related assets (html description and images), in what I call pointers, which point to the assets on the filesystem. I opted to use XML format for storing the data so it looks something like this:

// Path to pointer file
/home/files/app/needle/myApp/refrigerator/1/1.xml

// Example pointer
/home/files/app/file/myApp/refrigerator/1.png

Now, within the framework I had to override the save() methods so I could save the aforementioned assets using the NoSQL API. It was pretty easy, I just checked the parent calls and maintained the values coming into the methods, so they wouldn't break any chain logic (methods calling other methods with the same arguments), that I wasn't aware of. I also made my custom NoSQL API calls throw exceptions as the main save() call was wrapped in a try/catch block. The only thing you have to be careful of here, is determining whether your NoSQL assets are worth stopping the entire transaction. In my example, I had to figure out if uploading an image would break saving the rest of the form fields in the database (I opted to break the transaction).

I also had to alter the load() methods to retrieve the assets using the NoSQL API vs the standard model logic. As with the save methods, this wasn't too hard to do either. I just had to see what the parent classes were doing and not muck with any argument values.

When all was said and done, I was able to store images and html descriptions on the filesystem, with an xml file made up of pointers pointing to their location. So now I don't incur a database call every time I need an asset.

Some considerations (these may be included in other NoSQL solutions, I had to write my own):

• You will not be able to query for refrigerators that have images from your persistent store. You will have to write some logic in your application to pull in the assets from the NoSQL store.
• Backups: As you backup your persistent store data, you also need to backup your NoSQL data.
• Orphans: Now that your schema is unaware of any assets you may have, deleting a row from your persistent store will orphan an asset on the filesystem. So be sure your application has the logic to clean the NoSQL store when a row has been deleted.

I think I hit all the major hurdles I faced when implementing a NoSQL solution with an ORM, if you have any other questions feel free to hit me up.

-- Edit --

Responses to comments:

1. As I mentioned I didn't want to create a database connection and query just to get a path to an asset. I feel it's better to use a NoSQL solution for this type of information as there is really no reason run queries against this type of information (images or html descriptions).

2. Developing my own NoSQL solution was more of an ego challenge. At work there was a project to implement a custom NoSQL solution (had bad experiences with MogileFS), and to be frank, was poorly designed and poorly implemented. But rather than just point out the bad, I challenged myself to offer up a better solution, but for a side project. And because of the challenge aspect, I didn't research any already available NoSQL solutions, but in hindsight I probably should have.

I still think you can implement MongoDB or any NoSQL solution by overriding crud functions with the Model layer of your ORM, relatively easy. In fact, not only did I implement my NoSQL solution, I also added the ability to index data into SOLR (for full-text searching) during crud functions as well, so anything is possible.