Why is BigQuery so slow on non-large data sizes?

Question

We have found BigQuery to work great on data sets larger than 100M rows, where the \'initialization time\' doesn\'t really come into effect (or is negligible compared to the res

Answer 1

It's time spent on metadata/initiation, but actual execution time is very small. We have work in progress that will address this, but some of the changes are complicated and will take a while.

You can imagine that in its infancy, BigQuery could have central systems for managing jobs, metadata, etc. in a manner that performed very well for all N₀ entities using the service. Once you get to N₁ entities, however, it may be necessary to rearchitect some things to make them have as little latency as possible. For notification about new features--which is also where we would announce API improvements related to start-up latency--keep an eye on our release notes, which you can also subscribe to as an RSS feed.