Firestore, how to structure a “likedBy” query

Question

I\'m having a little trouble wrapping my head around how to best structure my (very simple) Firestore app. I have a set of users like this:

users: {
   \'A12         


        

Answer 1

There isn't a perfect solution for this problem, but there are alternatives you can do depending on what trade-offs you want.

The options: Overscan vs Underscan

Remember that Cloud Firestore only allows queries that scale independent of the total size of your dataset.

This can be really helpful in preventing you from building something that works in test with 10 documents, but blows up as soon as you go to production and become popular. Unfortunately, this type of problem doesn't fit that scalable pattern and the more profiles you have, and the more likes people create, the longer it takes to answer the query you want here.

The solution then is to find a one or more queries that scale and most closely represent what you want. There are 2 options I can think of that make trade-offs in different ways:

1. Overscan --> Do a broader query and then filter on the client-side
2. Underscan --> Do one or more narrower queries that might miss a few results.

Overscan

In the Overscan option, you're basically trading increased cost to get 100% accuracy.

Given your use-case, I imagine this might actually be your best option. Since the total number of profiles is likely orders of magnitude larger than the number of profiles an individual has liked, the increased cost of overscanning is probably inconsequential.

Simply select all profiles that match any other conditions you have, and then on the client side, filter out any that the user has already liked.

First, get all the profiles liked by the user:

var likedUsers = firebase.firestore().collection('users') .where('likedBy.C345','==',false)

Then get all users, checking against the first list and discarding anything that matches.

var allUsers = firebase.firestore().collection('users').get()

Depending on the scale, you'll probably want to optimize the first step, e.g. every time the user likes someone, update an array in a single document for that user for everyone they have liked. This way you can simply get a single document for the first step.

var likedUsers = firebase.firestore().collection('likedUsers') .doc('C345').get()

Since this query does scale by the size of the result set (by defining the result set to be the data set), Cloud Firestore can answer it without a bunch of hidden unscalable work. The unscalable part is left to you to optimize (with 2 examples above).

Underscan

In the Underscan option, you're basically trading accuracy to get a narrower (hence cheaper) set of results.

This method is more complex, so you probably only want to consider it if for some reason the liked to unliked ratio is not as I suspect in the Overscan option.

The basic idea is to exclude someone if you've definitely liked them, and accept the trade-off that you might also exclude someone you haven't yet liked - yes, basically a Bloom filter.

In each users profile store a map of true/false values from 0 to m (we'll get to what m is later), where everything is set to false initially.

When a user likes the profile, calculate the hash of the user's ID to insert into the Bloom filter and set all those bits in the map to true.

So let's say C345 hashes to 0110 if we used m = 4, then your map would look like:

likedBy: { 
   0: false,
   1: true,
   2: true,
   3: false }

Now, to find people you definitely haven't liked, you need use the same concept to do a query against each bit in the map. For any bit 0 to m that your hash is true on, query for it to be false:

var usersRef = firebase.firestore().collection('users')
.where('likedBy.1','==',false)

Etc. (This will get easier when we support OR queries in the future). Anyone who has a false value on a bit where your user's ID hashes to true definitely hasn't been liked by them.

Since it's unlikely you want to display ALL profiles, just enough to display a single page, you can probably randomly select a single one of the ID's hash bits that is true and just query against it. If you run out of profiles, just select another one that was true and restart.

Assuming most profiles are liked 500 or less times, you can keep the false positive ratio to ~20% or less using m = 1675.

There are handy online calculators to help you work out ratios of likes per profile, desired false positive ratio, and m, for example here.

Overscan - bonus

You'll quickly realize in the Overscan option that every time you run the query, the same profiles the user didn't like last time will be shown. I'm assuming you don't want that. Worse, all the ones the user liked will be early on in the query, meaning you'll end up having to skip them all the time and increase your costs.

There is an easy fix for that, use the method I describe on this question, Firestore: How to get random documents in a collection. This will enable you to pull random profiles from the set, giving you a more even distribution and reducing the chance of stumbling on lots of previously liked profiles.

Underscan - bonus

One problem I suspect you'll have with the Underscan option is really popular profiles. If someone is almost always liked, you might start exceeding the usefulness of a bloom filter if that profile has a size not reasonable to keep in a single document (you'll want m to be less than say 8000 to avoid running into per document index limits in Cloud Firestore).

For this problem, you want to combine the Overscan option just for these profiles. Using Cloud Functions, any profile that has more than x% of the map set to true gets a popular flag set to true. Overscan everyone on the popular flag and weave them into your results from the Underscan (remember to do the discard setup).