How to combine two RDD[String]s index-wise?

Question

I\'m working with Spark RDDs and created two idential length arrays, one is the hour of tweet, and the other is the text of a tweet. I\'m looking to combine these into one data

Answer 1

The answer by Ramesh Maharjan can work only under very specific assumptions:

• Both RDDs have the same number of partitions.
• Corresponding partitions have the same number of elements.

This trivial for ParallelCollectionRDD but it is hard or impossible to get in general.

It is much better, but costlier, to join:

split_time.zipWithIndex.map(_.swap).join(
  split_text.zipWithIndex.map(_.swap)
).values

or:

val split_time_with_index = split_time.zipWithIndex.map(_.swap)
val split_text_with_index = split_text.zipWithIndex.map(_.swap) 

val partitioner = new org.apache.spark.RangePartitioner(
  split_time.getNumPartitions, split_time
)

split_time.join(split_text, partitioner)