How to join on multiple columns in Pyspark?

前端 未结 2 1191
深忆病人
深忆病人 2021-01-30 16:29

I am using Spark 1.3 and would like to join on multiple columns using python interface (SparkSQL)

The following works:

I first register them as temp tables.

2条回答
  •  小蘑菇
    小蘑菇 (楼主)
    2021-01-30 16:59

    You should use & / | operators and be careful about operator precedence (== has lower precedence than bitwise AND and OR):

    df1 = sqlContext.createDataFrame(
        [(1, "a", 2.0), (2, "b", 3.0), (3, "c", 3.0)],
        ("x1", "x2", "x3"))
    
    df2 = sqlContext.createDataFrame(
        [(1, "f", -1.0), (2, "b", 0.0)], ("x1", "x2", "x3"))
    
    df = df1.join(df2, (df1.x1 == df2.x1) & (df1.x2 == df2.x2))
    df.show()
    
    ## +---+---+---+---+---+---+
    ## | x1| x2| x3| x1| x2| x3|
    ## +---+---+---+---+---+---+
    ## |  2|  b|3.0|  2|  b|0.0|
    ## +---+---+---+---+---+---+
    

提交回复
热议问题