How to use NOT IN clause in filter condition in spark

Question

I want to filter a column of an RDD source :

val source = sql(\"SELECT * from sample.source\").rdd.map(_.mkString(\",\"))
val destination = sql(\"select * from          


        

Answer 1

You can try like--

df.filter(~df.Dept.isin("30","20")).show() 

//This will list all the columns of df where Dept NOT IN 30 or 20