Trying to execute a spark sql query from a UDF

Question

I am trying to write a inline function in spark framework using scala which will take a string input, execute a sql statement and return me a String value

va         


        

Answer 1

You can't use sqlContext in your UDF - UDFs must be serializable to be shipped to executors, and the context (which can be thought of as a connection to the cluster) can't be serialized and sent to the node - only the driver application (where the UDF is defined, but not executed) can use the sqlContext.

Looks like your usecase (perform a select from table X per record in table Y) would better be accomplished by using a join.