Spark: Saving RDD in an already existing path in HDFS

Question

I am able to save the RDD output to HDFS with saveAsTextFile method. This method throws an exception if the file path already exists.

I have a use case

Answer 1

One possible solution, available since Spark 1.6, is to use DataFrames with text format and append mode:

val outputPath: String = ???

rdd.map(_.toString).toDF.write.mode("append").text(outputPath)