Spark dataframe filter

Question

val df = sc.parallelize(Seq((1,\"Emailab\"), (2,\"Phoneab\"), (3, \"Faxab\"),(4,\"Mail\"),(5,\"Other\"),(6,\"MSL12\"),(7,\"MSL\"),(8,\"HCP\"),(9,\"HCP12\"))).toDF(\"c1\"         


        

Answer 1

I used below to filter rows from dataframe and this worked form me.Spark 2.2

val spark = new org.apache.spark.sql.SQLContext(sc)    
val data = spark.read.format("csv").
          option("header", "true").
          option("delimiter", "|").
          option("inferSchema", "true").
          load("D:\\test.csv")   


import  spark.implicits._
val filter=data.filter($"dept" === "IT" )

OR

val filter=data.filter($"dept" =!= "IT" )