How to create encoder for Option type constructor, e.g. Option[Int]?

Question

Is it possible to use Option[_] member in a case class used with Dataset API? eg. Option[Int]

I tried to find an example but could not find

Answer 1

We only define implicits for a subset of the types we support in SQLImplicits. We should probably consider adding Option[T] for common T as the internal infrastructure does understand Option. You can workaround this by either creating a case class, using a Tuple or constructing the required implicit yourself (though this is using and internal API so may break in future releases).

implicit def optionalInt: org.apache.spark.sql.Encoder[Option[Int]] = org.apache.spark.sql.catalyst.encoders.ExpressionEncoder()

val ds = Seq(Some(1), None).toDS()

Answer 2

"Support for serializing other types will be added in future releases". Custom encoders aren't supported yet though obviously it's planned. You could try to implement the trait yourself, but there are certainly no official examples.

One option would be to use a Seq[Int] member and ensure it only has at most one value.