Problems to create DataFrame from Rows containing Option[T]

Question

I\'m migrating some code from Spark 1.6 to Spark 2.1 and struggling with the following issue:

This worked perfectly in Spark 1.6

import org.apache.spark.         


        

Answer 1

The error message is clear which says that Some is used when bigint is required

scala.Some is not a valid external type for schema of bigint

So you need to use Option combining with getOrElse so that we can define null when Option returns nullpointer. The following code should work for you

val sc = ss.sparkContext
val sqlContext = ss.sqlContext
val schema = StructType(Seq(StructField("i", LongType,nullable=true)))
val rows = sc.parallelize(Seq(Row(Option(1L) getOrElse(null))))
sqlContext.createDataFrame(rows,schema).show

I hope this answer is helpful