Manipulating a dataframe within a Spark UDF

Question

I have a UDF that filters and selects values from a dataframe, but it runs into \"object not serializable\" error. Details below.

Suppose I have a dataframe df1 that has

Answer 1

1) No, you can only use plain scala code within UDFs

2) If you interpreted your code correctly, you can achieve your goal with:

df2
  .join(
    df1.select($"ID",y_list.foldLeft(lit(0))(_ + _).as("Result")),Seq("ID")
  )