Extreme Sharding: One SQLite Database Per User

Question

I\'m working on a web app that is somewhere between an email service and a social network. I feel it has the potential to grow really big in the future, so I\'m concerned about

Answer 1

If your data is this easy to shard, why not just use a standard database engine, and if you scale large enough that the DB becomes the bottleneck, shard the database, with different users in different instances? The effect is the same, but you're not using scores of tiny little databases.

In reality, you probably have at least some shared data that doesn't belong to any single user, and you probably frequently need to access data for more than one user. This will cause problems with either system, though.

Answer 2

I am considering this same architecture as I basically wanted to use the server side SQLLIte databases as backup and synching copy for clients. My idea for querying across all the data is to use Sphinx for full-text search and run Hadoop jobs from flat dumps of all the data to Scribe and then expose the results as webservies. This post gives me some pause for thought however, so I hope people will continue to respond with their opinion.

Answer 3

Having one database per user would make it really easy to restore individual users data of course, but as @John said, schema changes would require some work.

Not enough to make it hard, but enough to make it non-trivial.

Answer 4

One possible problem is that having one database for each user will use disk space and RAM very inefficiently, and as the user base grows the benefit of using a light and fast database engine will be lost completely.

A possible solution to this problem is to create "minishards" consisting of maybe 1024 SQLite databases housing up to 100 users each. This will be more efficient than the DB per user approach, because data is packed more efficiently. And lighter than the Innodb database server approach, because we're using Sqlite.

Concurrency will also be pretty good, but queries will be less elegant (shard_id yuckiness). What do you think?

Answer 5

If you're creating a separate database for each user, it sounds like you're not setting up relationships... so why use a relational database at all?

Answer 6

http://freshmeat.net/projects/sphivedb

SPHiveDB is a server for sqlite database. It use JSON-RPC over HTTP to expose a network interface to use SQLite database. It supports combining multiple SQLite databases into one file. It also supports the use of multiple files. It is designed for the extreme sharding schema -- one SQLite database per user.