Why inconsistent results using subtraction in reduce?

Question

Given the following:

val rdd = List(1,2,3)

I assumed that rdd.reduce((x,y) => (x - y)) would return -4 (i.e. <

Answer 1

From the RDD source code (and docs):

/**
* Reduces the elements of this RDD using the specified commutative and
* associative binary operator.
*/
def reduce(f: (T, T) => T): T

reduce is a monoidal reduction, thus it assumes the function is commutative and associative, meaning that the order of applying it to the elements is not guaranteed.

Obviously, your function (x,y)=>(x-y) isn't commutative nor associative.

In your case, the reduce might have been applied this way:

3 - (2 - 1) = 2

or

1 - (2 - 3) = 2