Filter spark/scala dataframe if column is present in set

Question

I\'m using Spark 1.4.0, this is what I have so far:

data.filter($\"myColumn\".in(lit(\"A\"), lit(\"B\"), lit(\"C\"), ...))

The function lit

Answer 1

Spark 1.4 or older:

val validValues = Set("A", "B", "C").map(lit(_))
data.filter($"myColumn".in(validValues.toSeq: _*))

Spark 1.5 or newer:

val validValues = Set("A", "B", "C")
data.filter($"myColumn".isin(validValues.toSeq: _*))