Google Firestore: Query on substring of a property value (text search)

Question

I am looking to add a simple search field, would like to use something like

collectionRef.where(\'name\', \'contains\', \'searchTerm\')

I tried

Answer 1

Per the Firestore docs, Cloud Firestore doesn't support native indexing or search for text fields in documents. Additionally, downloading an entire collection to search for fields client-side isn't practical.

Third-party search solutions like Algolia and Elastic Search are recommended.

Answer 2

I just had this problem and came up with a pretty simple solution.

String search = "ca";
Firestore.instance.collection("categories").orderBy("name").where("name",isGreaterThanOrEqualTo: search).where("name",isLessThanOrEqualTo: search+"z")

The isGreaterThanOrEqualTo lets us filter out the beginning of our search and by adding a "z" to the end of the isLessThanOrEqualTo we cap our search to not roll over to the next documents.

Answer 3

With Firestore you can implement a full text search but it will still cost more reads than it would have otherwise, and also you'll need to enter and index the data in a particular way, So in this approach you can use firebase cloud functions to tokenise and then hash your input text while choosing a linear hash function h(x) that satisfies the following - if x < y < z then h(x) < h (y) < h(z). For tokenisation you can choose some lightweight NLP Libraries in order to keep the cold start time of your function low that can strip unnecessary words from your sentence. Then you can run a query with less than and greater than operator in Firestore. While storing your data also, you'll have to make sure that you hash the text before storing it, and store the plain text also as if you change the plain text the hashed value will also change.

Answer 4

While Kuba's answer is true as far as restrictions go, you can partially emulate this with a set-like structure:

{
  'terms': {
    'reebok': true,
    'mens': true,
    'tennis': true,
    'racket': true
  }
}

Now you can query with

collectionRef.where('terms.tennis', '==', true)

This works because Firestore will automatically create an index for every field. Unfortunately this doesn't work directly for compound queries because Firestore doesn't automatically create composite indexes.

You can still work around this by storing combinations of words but this gets ugly fast.

You're still probably better off with an outboard full text search.

Answer 5

The selected answer only works for exact searches and is not natural user search behavior (searching for "apple" in "Joe ate an apple today" would not work).

I think Dan Fein's answer above should be ranked higher. If the String data you're searching through is short, you can save all substrings of the string in an array in your Document and then search through the array with Firebase's array_contains query. Firebase Documents are limited to 1 MiB (1,048,576 bytes) (Firebase Quotas and Limits) , which is about 1 million characters saved in a document (I think 1 character ~= 1 byte). Storing the substrings is fine as long as your document isn't close to 1 million mark.

Example to search user names:

Step 1: Add the following String extension to your project. This lets you easily break up a string into substrings. (I found this here).

extension String {

var length: Int {
    return count
}

subscript (i: Int) -> String {
    return self[i ..< i + 1]
}

func substring(fromIndex: Int) -> String {
    return self[min(fromIndex, length) ..< length]
}

func substring(toIndex: Int) -> String {
    return self[0 ..< max(0, toIndex)]
}

subscript (r: Range<Int>) -> String {
    let range = Range(uncheckedBounds: (lower: max(0, min(length, r.lowerBound)),
                                        upper: min(length, max(0, r.upperBound))))
    let start = index(startIndex, offsetBy: range.lowerBound)
    let end = index(start, offsetBy: range.upperBound - range.lowerBound)
    return String(self[start ..< end])
}

Step 2: When you store a user's name, also store the result of this function as an array in the same Document. This creates all variations of the original text and stores them in an array. For example, the text input "Apple" would creates the following array: ["a", "p", "p", "l", "e", "ap", "pp", "pl", "le", "app", "ppl", "ple", "appl", "pple", "apple"], which should encompass all search criteria a user might enter. You can leave maximumStringSize as nil if you want all results, however, if there is long text, I would recommend capping it before the document size gets too big - somewhere around 15 works fine for me (most people don't search long phrases anyway).

func createSubstringArray(forText text: String, maximumStringSize: Int?) -> [String] {

    var substringArray = [String]()
    var characterCounter = 1
    let textLowercased = text.lowercased()

    let characterCount = text.count
    for _ in 0...characterCount {
        for x in 0...characterCount {
            let lastCharacter = x + characterCounter
            if lastCharacter <= characterCount {
                let substring = textLowercased[x..<lastCharacter]
                substringArray.append(substring)
            }
        }
        characterCounter += 1

        if let max = maximumStringSize, characterCounter > max {
            break
        }
    }

    print(substringArray)
    return substringArray
}

Step 3: You can use Firebase's array_contains function!

[yourDatabasePath].whereField([savedSubstringArray], arrayContains: searchText).getDocuments....

Answer 6

As of today, there are basically 3 different workarounds, which were suggested by the experts, as answers to the question.

I have tried them all. I thought it might be useful to document my experience with each one of them.

Method-A: Using: (dbField ">=" searchString) & (dbField "<=" searchString + "\uf8ff")

Suggested by @Kuba & @Ankit Prajapati

.where("dbField1", ">=", searchString)
.where("dbField1", "<=", searchString + "\uf8ff");

A.1 Firestore queries can only perform range filters (>, <, >=, <=) on a single field. Queries with range filters on multiple fields are not supported. By using this method, you can't have a range operator in any other field on the db, e.g. a date field.

A.2. This method does NOT work for searching in multiple fields at the same time. For example, you can't check if a search string is in any of the fileds (name, notes & address).

Method-B: Using a MAP of search strings with "true" for each entry in the map, & using the "==" operator in the queries

Suggested by @Gil Gilbert

document1 = {
  'searchKeywordsMap': {
    'Jam': true,
    'Butter': true,
    'Muhamed': true,
    'Green District': true,
    'Muhamed, Green District': true,
  }
}

.where(`searchKeywordsMap.${searchString}`, "==", true);

B.1 Obviously, this method requires extra processing every time data is saved to the db, and more importantly, requires extra space to store the map of search strings.

B.2 If a Firestore query has a single condition like the one above, no index needs to be created beforehand. This solution would work just fine in this case.

B.3 However, if the query has another condition, e.g. (status === "active",) it seems that an index is required for each "search string" the user enters. In other words, if a user searches for "Jam" and another user searches for "Butter", an index should be created beforehand for the string "Jam", and another one for "Butter", etc. Unless you can predict all possible users' search strings, this does NOT work - in case of the query has other conditions!

.where(searchKeywordsMap["Jam"], "==", true); // requires an index on searchKeywordsMap["Jam"]
.where("status", "==", "active");

**Method-C: Using an ARRAY of search strings, & the "array-contains" operator

Suggested by @Albert Renshaw & demonstrated by @Nick Carducci

document1 = {
  'searchKeywordsArray': [
    'Jam',
    'Butter',
    'Muhamed',
    'Green District',
    'Muhamed, Green District',
  ]
}

.where("searchKeywordsArray", "array-contains", searchString); 

C.1 Similar to Method-B, this method requires extra processing every time data is saved to the db, and more importantly, requires extra space to store the array of search strings.

C.2 Firestore queries can include at most one "array-contains" or "array-contains-any" clause in a compound query.

General Limitations:

1. None of these solutions seem to support searching for partial strings. For example, if a db field contains "1 Peter St, Green District", you can't search for the string "strict."
2. It is almost impossible to cover all possible combinations of expected search strings. For example, if a db field contains "1 Mohamed St, Green District", you may NOT be able to search for the string "Green Mohamed", which is a string having the words in a different order than the order used in the db field.

There is no one solution that fits all. Each workaround has its limitations. I hope the information above can help you during the selection process between these workarounds.

For a list of Firestore query conditions, please check out the documentation https://firebase.google.com/docs/firestore/query-data/queries.

I have not tried https://fireblog.io/blog/post/firestore-full-text-search, which is suggested by @Jonathan.