How to split dataset to two datasets with unique and duplicate rows each?

Question

I want to take duplicate records in a Spark scala Dataframe. for example, I want to take duplicate values based on 3 columns like \"id\", \"name\", \"age\".condition part co

Answer 1

You need to give comma separated col names.

col1 ..col2 should be of string type.
     val window= Window.partitionBy(col1,col2,..)


    findDuplicateRecordsDF.withColumn("count", count("*")
          .over(window)
          .where($"count">1)
          .show()