Convert value depending on a type in SparkSQL via case matching of type

Question

Is it possible to match a parametric type in Scala? Lets say I have a function that receives two parameters: a value and a type. I would like to us

Answer 1

Turns out that DecimalType only pattern matches with zero arguments:

  case DecimalType() => ...

If you need the precision and scale, you must define the type of the case and manually extract them:

datatype match {
  case dt: DecimalType =>
    val precision = dt.precision
    val scale = dt.scale
    ...

Answer 2

The problem is the use of the return in your code. You said you use this code snippet in a function somewhere. What is the return type of that function? Obviously, you intend that sometimes it is Integer, sometimes String, sometimes BigDecimal; but if you use return, it will look to the type of the returned object to determine the return type of the function. In general, you should strongly avoid using return in Scala code. The last evaluated value in the function body is returned. The only case for using a return is when you want to force returning a value somewhere else in the function body. But still, a better way would be to save the return object in a variable and just evaluate that variable in the last line of your function body. And never use return!

Without return it works

scala> val datatype = DecimalType(10, 2)
datatype: org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

scala> val value = BigDecimal(10)
value: scala.math.BigDecimal = 10

scala> datatype match {case DecimalType(_,_) => value}
res150: scala.math.BigDecimal = 10

** Problems with return **

scala> def test = {datatype match {case DecimalType(_,_) => return value}}
<console>:138: error: method test has return statement; needs result type
       def test = {datatype match {case DecimalType(_,_) => return value}}

scala> def test:BigDecimal = {datatype match {case DecimalType(_,_) => return value}}
test: BigDecimal

scala> def test:DataType = {datatype match {case DecimalType(_,_) => return value}}
<console>:138: error: type mismatch;
 found   : scala.math.BigDecimal
 required: org.apache.spark.sql.types.DataType
       def test:DataType = {datatype match {case DecimalType(_,_) => return value}}

scala> def test3 = {datatype match {case DecimalType(_,_) => value}}
test3: scala.math.BigDecimal

Answer 3

Could be something specific to the code I'm working on, or perhaps it varies depending on the SQL vendor, but I found that DecimalType doesn't have a single underlying type. Sometimes I get a spark Decimal and other times I get a java BigDecimal. If I try to getAs[Decimal] when it's a BigDecimal, I get an exception. If I try to getAs[BigDecimal] when it's a Decimal, I get an exception.

To handle this I had to do some more sniffing after matching DecimalType:

  case d: DecimalType =>
    // Oddly a column that matches to DecimalType can be of several different
    // class types and trying to getAs[Decimal] when it's a BigDecimal and/or
    // trying to getAs[BigDecimal] when the column is a Decimal results in an
    // exception, so make the right decision based on the instance class.
    val decimal = row.get(index) match {
      case bigDecimal: java.math.BigDecimal => Decimal(bigDecimal)
      case decimal: Decimal => decimal
      case _ => throw(
        new RuntimeException("Encountered unexpected decimal type")
      )
    }

From there you can do whatever you need to do knowing that decimal is of type Decimal.