What is the best way to associate a file with a piece of data?

Question

I have an application that creates records in a table (rocket science, I know). Users want to associate files (.doc, .xls, .pdf, etc...) to a single record in the table.

Answer 1

And now for the completely off the wall suggestion - you could consider storing the binaries as attachments in a CouchDB document database. This would avoid the file name collision issues as you would use a generated UID as each document ID (which you what you would store in your RDBMS), and the actual attachment's file name is kept with the document.

If you are building a web-based system, then the fact that CouchDB uses REST over HTTP could also be leveraged. And, there's also the replication facilities that could prove of use.

Of course, CouchDB is still in incubation, although there are some who are already using it 'in the wild'.

Answer 2

Store the paths in the database. This keeps your database from bloating, and also allows you to separately back up the external files. You can also relocate them more easily; just move them to a new location and then UPDATE the database.

One additional thing to keep in mind: In order to use most of the filetypes you mentioned, you'll end up having to:

• Query the database to get the file contents in a blob
• Write the blob data to a disk file
• Launch an application to open/edit/whatever the file you just created
• Read the file back in from disk to a blob
• Update the database with the new content

All that as opposed to:

• Read the file path from the DB
• Launch the app to open/edit/whatever the file

I prefer the second set of steps, myself.

Answer 3

I would try to store it all in the database. Haven't done it. But if not. There are a small risk that file names get out of sync with files on the disk. Then you have a big problem.

Answer 4

How well you can store binaries, or BLOBs, in a database will be highly dependant on the DBMS you are using.

If you store binaries on the file system, you need to consider what happens in the case of file name collision, where you try and store two different files with the same name - and if this is a valid operation or not. So, along with the reference to where the file lives on the file system, you may also need to store the original file name.

Also, if you are storing a large amount of files, be aware of possible performance hits of storing all your files in one folder. (You didn't specify your operating system, but you might want to look at this question for NTFS, or this reference for ext3.)

We had a system that had to store several thousands of files on the file system, on a file system where we were concerned about the number of files in any one folder (it may have been FAT32, I think).

Our system would take a new file to be added, and generate an MD5 checksum for it (in hex). It would take the first two characters and make that the first folder, the next two characters and make that the second folder as a sub-folder of the first folder, and then the next two as the third folder as a sub-folder of the second folder.

That way, we ended up with a three-level set of folders, and the files were reasonably well scattered so no one folder filled up too much.

If we still had a file name collision after that, then we would just add "_n" to the file name (before the extension), where n was just an incrementing number until we got a name that didn't exist (and even then, I think we did atomic file creation, just to be sure).

Of course, then you need tools to do the occasional comparison of the database records to the file system, flagging any missing files and cleaning up any orphaned ones where the database record no longer exists.

Answer 5

You should only store files in the database if you're reasonably sure you know that the sizes of those files aren't going to get out of hand.

I use our database to store small banner images, which I always know what size they're going to be. Your database will store a pointer to the data inside a row and then plunk the data itself somewhere else, so it doesn't necessarily impact speed.

If there are too many unknowns though, using the filesystem is the safer route.

Answer 6

Use the database for data and the filesystem for files. Simply store the file path in the database.

In addition, your webserver can probably serve files more efficiently than you application code will do (in order to stream the file from the DB back to the client).