How to create a DataFrame from a text file in Spark

后端 未结 8 1047
滥情空心
滥情空心 2021-01-31 19:03

I have a text file on HDFS and I want to convert it to a Data Frame in Spark.

I am using the Spark Context to load the file and then try to generate individual columns f

8条回答
  •  [愿得一人]
    2021-01-31 19:49

    val df = spark.read.textFile("abc.txt")
    
    case class Abc (amount:Int, types: String, id:Int)  //columns and data types
    
    val df2 = df.map(rec=>Amount(rec(0).toInt, rec(1), rec(2).toInt))
    rdd2.printSchema
    

    root
     |-- amount: integer (nullable = true)
     |-- types: string (nullable = true)
     |-- id: integer (nullable = true)
    

提交回复
热议问题