How to use COGROUP for large datasets

Question

I have two rdd\'s namely val tab_a: RDD[(String, String)] and val tab_b: RDD[(String, String)] I\'m using cogroup for tho

Answer 1

When you use collect() you are basically telling spark to move all the resulting data back to the master node, which can easily produce a bottleneck. You are no longer using Spark at that point, just a plain array in a single machine.

To trigger computation just use something that requires the data at every node, that's why executors live on top of a distributed file system. For instance saveAsTextFile().

Here are some basic examples.

Remember, the entire objective here (that is, if you have big data) is to move the code to your data and compute there, not to bring all the data to the computation.

Answer 2

TL;DR Don't collect.

To run this code safely, without additional assumptions (on average requirements for worker nodes might be significantly smaller), every node (driver and each executor) would require memory significantly exceeding total memory requirements for all data.

If you were to run it outside Spark you would need only one node. Therefore Spark provides no benefits here.

However if you skip collect.toArray and make some assumptions about data distribution you might run it just fine.