How to get the values in DataFrame with the correct DataType?

Question

When I tried to get some values in a DataFrame, like:

df.select(\"date\").head().get(0) // type: Any

The result type is Any<

Answer 1

You can call the generic getAs method as getAs[Int](columnIndex), getAs[String](columnIndex) or use specific methods like getInt(columnIndex), getString(columnIndex).

Link to the Scaladoc for org.apache.spark.sql.Row.

Answer 2

Scala is a statically typed language. so the get method defined on the Row can only return values with a single type because the return type of the get method is Any. It cannot return Int for one call and a String for another.

you should be calling the getInt, getDate and other get methods provided for each type. Or the getAs method in which you can pass the type as a parameter (for example row.getAs[Int](0)).

As mentioned in the comments other options are

• use Dataset instead of a DataFrame.
• use Spark SQL