Write single CSV file using spark-csv

前端 未结 13 1858
心在旅途
心在旅途 2020-11-22 08:43

I am using https://github.com/databricks/spark-csv , I am trying to write a single CSV, but not able to, it is making a folder.

Need a Scala function which will take

13条回答
  •  伪装坚强ぢ
    2020-11-22 09:04

    by using Listbuffer we can save data into single file:

    import java.io.FileWriter
    import org.apache.spark.sql.SparkSession
    import scala.collection.mutable.ListBuffer
        val text = spark.read.textFile("filepath")
        var data = ListBuffer[String]()
        for(line:String <- text.collect()){
          data += line
        }
        val writer = new FileWriter("filepath")
        data.foreach(line => writer.write(line.toString+"\n"))
        writer.close()
    

提交回复
热议问题