Convert Row to map in spark scala

Question

I have a row from a data frame and I want to convert it to a Map[String, Any] that maps column names to the values in the row for that column.

Is there an easy way to do

Answer 1

Let's say you have a row without structure information and the column header as an array.

val rdd = sc.parallelize( Seq(Row("test1", "val1"), Row("test2", "val2"), Row("test3", "val3"), Row("test4", "val4")) )
rdd.collect.foreach(println)

val sparkFieldNames = Array("col1", "col2")

val mapRDD = rdd.map(
  r => sparkFieldNames.zip(r.toSeq).toMap
)

mapRDD.collect.foreach(println)