How to Join Multiple Columns in Spark SQL using Java for filtering in DataFrame

Question

• DataFrame a = contains column x,y,z,k

• DataFrame b = contains column x,y,a

  a.join(b,
  
          

Answer 1

If you want to use Multiple columns for join, you can do something like this:

a.join(b,scalaSeq, joinType)

You can store your columns in Java-List and convert List to Scala seq. Conversion of Java-List to Scala-Seq:

scalaSeq = JavaConverters.asScalaIteratorConverter(list.iterator()).asScala().toSeq();

Example: a = a.join(b, scalaSeq, "inner");

Note: Dynamic columns will be easily supported in this way.