How to validate date format in a dataframe column in spark scala

Question

I have a dataframe with one DateTime column and many other columns.

All I wanted to do is parse this DateTime column value and check if the format is \"yyyy-MM

Answer 1

Here we define a function for checking whether a String is compatible with your format requirements, and we partition the list into compatible/non pieces. The types are shown with full package names, but you should use import statements, of course.

val fmt = "yyyy-MM-dd HH:mm:ss"
val df = java.time.format.DateTimeFormatter.ofPattern(fmt)
def isCompatible(s: String) = try {
  java.time.LocalDateTime.parse(s, df)
  true
} catch {
  case e: java.time.format.DateTimeParseException => false
}
val dts = Seq("2016-11-07 15:16:17", "2016-11-07 24:25:26")
val yesNo = dts.partition { s => isCompatible(s) }
println(yesNo)