Writing a RDD to a csv

Question

I have a RDD which is of the form

org.apache.spark.rdd.RDD[(String, Array[String])]

I want to write this into a csv file. Please suggest m

Answer 1

You can try:

myrdd.map(a => a._1 + "," + a._2.mkString(",")).saveAsTextFile

Answer 2

The other answer doesn't cater for escaping. Perhaps this more general solution?

import au.com.bytecode.opencsv.CSVWriter
import java.io.StringWriter
import scala.collection.JavaConversions._
val toCsv = (a: Array[String]) => {
  val buf = new StringWriter
  val writer = new CSVWriter(buf)
  writer.writeAll(List(a))
  buf.toString.trim
}
rdd.map(t => Array(t._1) ++ t._2)
   .map(a => toCsv(a))
   .saveAsTextFile(dest)