Firebase: queries on large datasets

Question

I\'m using Firebase to store user profiles. I tried to put the minimum amount of data in each user profile (following the good practices advised in the documentation about s

Answer 1

The core problem here isn't the query or the size of the data, it's simply the time required to warm the data into memory (i.e. load it from disk) when it's not being frequently queried. It's likely to be only a development issue, as in production this query would likely be a more frequently used asset.

But if the goal is to improve performance on initial load, the only reasonable answer here is to query on less data. 150MB is significant. Try copying a 150MB file between computers over a wireless network and you'll have some idea what it's like to send it over the internet, or to load it into memory from a file server.

A lot here depends on the use case, which you haven't included.

Assuming you have fairly standard search criteria (e.g. you search on email addresses), you can use indices to store email addresses separately to reduce the data set for your query.

/search_by_email/$user_id/<email address>

Now, rather than 50k per record, you have only the bytes to store the email address per records--a much smaller payload to warm into memory.

Assuming you're looking for robust search capabilities, the best answer is to use a real search engine. For example, enable private backups and export to BigQuery, or go with ElasticSearch (see Flashlight for an example).