Storing & accessing up to 10 million files in Linux

Question

I\'m writing an app that needs to store lots of files up to approx 10 million.

They are presently named with a UUID and are going to be around 4MB each but always the sa

Answer 1

Modern filesystems will let you store 10 million files all in the same directory if you like. But tools (ls and its friends) will not work well.

I'd recommend putting a single level of directories, a fixed number, perhaps 1,000 directories, and putting the files in there (10,000 files is tolerable to the shell, and "ls").

I've seen systems which create many levels of directories, this is truly unnecessary and increases inode consumption and makes traversal slower.

10M files should not really be a problem either, unless you need to do bulk operations on them.

I expect you will need to prune old files, but something like "tmpwatch" will probably work just fine with 10M files.