Efficient Count Distinct with Apache Spark

Question

100 million customers click 100 billion times on the pages of a few web sites (let\'s say 100 websites). And the click stream is available to you in a large dataset.

Usi

Answer 1

Spark 2.0 added ApproxCountDistinct into dataframe and SQL APIs:

https://databricks.com/blog/2016/05/19/approximate-algorithms-in-apache-spark-hyperloglog-and-quantiles.html

https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/sql/functions.html#approxCountDistinct(org.apache.spark.sql.Column)

Answer 2

Df.select(approx_count_distinct("col_name",0.1))

0.1 is the parameter which is saying maximum estimated error allowed. You can see much great performance with large data set.