Error: not found: value lit/when - spark scala

Question

I am using scala, spark, IntelliJ and maven.

I have used below code :

val joinCondition = when($\"exp.fnal_expr_dt\" >= $\"exp.nonfnal_expr_dt\",         


        

Answer 1

Let's consider the following context :

val spark : SparkSession = _ // or val sqlContext: SQLContext = new SQLContext(sc) for 1.x
val list: DataFrame = ???

To use when and lit, you'll need to import the proper functions :

import org.apache.spark.sql.functions.{col, lit, when}

Now you can use them as followed :

list.select(when(col("column_name").isNotNull, lit(1)))

Now you can use lit also in your code :

val score = list.withColumn("scr", lit(0))