How to return a case class when using Spark High Order Functions?

Question

I am trying to use Spark transform function in order to transform the items of an array from type ClassA into ClassB as shown below:

Answer 1

The transform expression is relational and doesn't know anything about case classes ClassA and ClassB. The only way you have AFAIK would be to register an UDF so you can use your structure (or inject functions) but you would also have to deal with a "Row" encoded value instead of ClassA (SparkSQL is all about encoding :) ) like so :

sparkSession.udf.register("toB", (a: Row) => ClassB(a.getAs[String]("a"), a.getAs[String]("b")))

df.withColumn("ClassB", expr("transform(ClassA, c -> toB(c))")).show(false)

Side note: Naming your column "ClassA" might be confusing since transform is reading the column, not the type.