Spark Dataset select with typedcolumn

Question

Looking at the select() function on the spark DataSet there are various generated function signatures:

(c1: TypedColumn[MyClass, U1],c2: TypedColumn         


        

Answer 1

In the Scala DSL for select, there are many ways to identify a Column:

• From a symbol: 'name
• From a string: $"name" or col(name)
• From an expression: expr("nvl(name, 'unknown') as renamed")

To get a TypedColumn from Column you simply use myCol.as[T].

For example: ds.select(col("name").as[String])