RDD.union vs SparkContex.union

Question

What is the difference between

reduce(lambda x,y: x.union(y), myRDDlist)

which calls RDD.union and

sc.union(myRDDlist)


        

Answer 1

SparkContext.union and RDD.union are equivalent, if you have two RDDs.

Reducing over a list of RDDs and calling RDD.union will result in several nested UnionRDDs (referencing each other), where the call to SparkContext.union will result in only a single UnionRDD.