How to Join Mutiple Columns in Spark SQL using Java for filtering in DataFrame

问题:

• DataFrame a = contains column x,y,z,k

• DataFrame b = contains column x,y,a

  a.join(b,<condition to use in java to use x,y >) ???  

I tried using

a.join(b,a.col("x").equalTo(b.col("x")) && a.col("y").equalTo(b.col("y"),"inner") 

But Java is throwing error saying && is not allowed.

Can someone help ? thanks

回答1:

Spark SQL provides a group of methods on Column marked as java_expr_ops which are designed for Java interoperability. It includes and (see also or) method which can be used here:

a.col("x").equalTo(b.col("x")).and(a.col("y").equalTo(b.col("y")) 

文章来源: How to Join Mutiple Columns in Spark SQL using Java for filtering in DataFrame