What are docValues in Solr? When should I use them?

Question

So, I have read multiple sources that try to explain what \'docValues\' are in Solr, but I don\'t seem to understand when I should use them, especially in relation to indexed vs

Answer 1

Use cases of DocValues are already explained by @Persimmonium and are pretty clear. they are good for faceting and sorting and such fancy stuff in the IR world.

What are docValue and why they are there ? docValue is nothing but a way to build a forward index so that documents point to values. they are built to overcome the limitations of FieldCache by providing a document to value mapping built at index time and they store values in a column based fashion and it does all the heavyweight lifting during document indexing.

What docvalues are:

NRT-compatible: These are per-segment datastructures built at index-time and designed to be efficient for the use case where data is changing rapidly.

Basic query/filter support: You can do basic term, range, etc queries on docvalues fields without also indexing them, but these are constant-score only and typically slower. If you care about performance and scoring, index the field too.

Better compression than fieldcache: Docvalues fields compress better than fieldcache, and "insanity" is impossible.

Able to store data outside of heap memory: You can specify a different docValuesFormat on the fieldType (docValuesFormat="Disk") to only load minimal data on the heap, keeping other data structures on disk.

What docvalues are not:

Not a replacement for stored fields: These are unrelated to stored fields in every way and instead datastructures for search (sort/facet/group/join/scoring).

Use case to use with Lucene docValues this way.

    public Bits getDocsWithField(FieldInfo field) throws IOException {
  switch(field.getDocValuesType()) {
    case SORTED_SET:
      return DocValues.docsWithValue(getSortedSet(field), maxDoc);
    case SORTED_NUMERIC:
      return DocValues.docsWithValue(getSortedNumeric(field), maxDoc);
    case SORTED:
      return DocValues.docsWithValue(getSorted(field), maxDoc);
    case BINARY:
      BinaryEntry be = binaries.get(field.number);
      return getMissingBits(be.missingOffset);
    case NUMERIC:
      NumericEntry ne = numerics.get(field.number);
      return getMissingBits(ne.missingOffset);
    default:
      throw new AssertionError();
  }
}

Answer 2

Due to the way they are stored and accessed, they will speed up some operations, like sorting, faceting etc.

Besides, they are mandatory for using some features: streaming expressions, in place updates...

So, if in doubt:

1. if you don't have a big index, and size is not a problem, just enable them
2. if you do have a huge index, or indexing perf is critical, look into them more carefully and pick which fields to enable them on

Answer 3

What are docValues in Solr ?

Doc values can be explained as Lucene's column-stride field value storage or simply its an uninverted index or forward index.

To illustrate with json:

• row-oriented (stored fields)

  
  {
  'doc1': {'A':1, 'B':2, 'C':3},
  'doc2': {'A':2, 'B':3, 'C':4},
  'doc3': {'A':4, 'B':3, 'C':2}
  }
  

• column-oriented (docValues)

  
  {
  'A': {'doc1':1, 'doc2':2, 'doc3':4},
  'B': {'doc1':2, 'doc2':3, 'doc3':3},
  'C': {'doc1':3, 'doc2':4, 'doc3':2}
  }
  

Purpose of DocValues ?

Stored fields store all field values for one document together in a row-stride fashion. In retrieval, all field values are returned at once per document, so that loading the relevant information about a document is very fast.

However, if you need to scan a field (for faceting/sorting/grouping/highlighting) it will be a slow process, as you will have to iterate through all the documents and load each document's fields per iteration resulting in disk seeks.

For example, sorting, when all the matched documents are found, Lucene need to get the value of a field of each of them. Similarly the faceting engine, for example, must look up each term that appears in each document that will make up the result set and pull the document IDs in order to build the facet list.

Now this problem can be approached in two ways:

• Using existing stored fields. In that case if you start sorting/aggregating on a given field, data will be lazily un-inverted and put into a fieldCache at search time so that you can access values given a document ID. This process is very CPU and I/O intensive.
• DocValues are quite fast to access at search time, since they are stored column-stride such that only the value for that one field needs to be decoded per hit. This approach promises to relieve some of the memory requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.

Like inverted index docvalues are serialized to disk in that case we can rely on the OS’s file system cache to manage memory instead of retaining structures on the JVM heap.

When should I use them ?

For all the reasons discussed above. If you are in a low-memory environment, or you don’t need to index a field, DocValues are perfect for faceting/grouping/filtering/sorting/function queries. They also have the potential for increasing the number of fields you can facet/group/filter/sort on without increasing your memory requirements. I've been using docvalues in production Solr for sorting and faceting and have seen a huge improvement in performance of these queries.