Why inconsistent results using subtraction in reduce?

前端 未结 3 1227
既然无缘
既然无缘 2021-01-19 13:50

Given the following:

val rdd = List(1,2,3)

I assumed that rdd.reduce((x,y) => (x - y)) would return -4 (i.e. <

3条回答
  •  时光说笑
    2021-01-19 14:36

    As aforementioned by @TzachZohar the function must satisfy the two properties so that the parallel computation is sound; by collecting the rdd, reduce relaxes the properties required in the function, and so it produces the result from a sequential (non parallel) computation, namely,

    val rdd = sc.parallelize(1 to 3)
    
    rdd.collect.reduce((x,y) => (x-y))
    Int = -4
    

提交回复
热议问题