adding a new column using withColumn from a lookup table dynamically

Question

I am using spark-sql-2.4.1v with Java 8. I have a scenario where I need to dynamically add a column from a look up table.

I have data frame with columns A, B, C , ..., X,

Answer 1

In Scala, I would do like this

val substitueMapping: Map[String, String] = ??? //this is your substitute map, this is small as it contains columns and their null substitutes

val df = ??? //this is your main dataframe 

val substitutedDf = substituteMapping.keys().foldLeft(df)((df, k) => {
    df.withColumn(k, when(col(k).isNull, col(substituteMapping(k))).otherwise(col(k)))
    //do approproate casting in above which you have done in post
})

I think foldLeft is not there in Java 8, you can emulate the same by modifying a variable repeatedly and doing iteration on substituteMapping.