BigQuery error “Response too large to return” when using COUNT(DISTINCT …)

Question

I have a dataset with ~20M rows and I\'m observing the following behavior.

The query below returns the error \"Response too large to return\". The \'id\' field is sh

Answer 1

It's not the result size that is causing this response, it's the intermediate size of the data generated by your COUNT DISTINCT query.

Note: COUNT DISTINCT returns a statistical approximation after 1000 values - you can alter the approximation by choosing a particular value for the limit in which DISTINCT will return an approximation.. such as: COUNT(DISTINCT your_field, 500)

See: https://developers.google.com/bigquery/docs/query-reference#aggfunctions

This behavior is due to the design of BigQuery, which makes it so quick: data is queried via separate nodes, and results are aggregated at mixers. A COUNT will tally the total number of results and combine the answer, but COUNT DISTINCT needs to keep track of potentially millions of separate sums, and then combine those values later. Therefore a COUNT DISTINCT can create a lot of data, and could potentially be over internal maximum for individual nodes.

Note also that currently, BigQuery LIMIT clauses are applied after the entire result set is determined.