Join two DataFrames where the join key is different and only select some columns

Question

What I would like to do is:

Join two DataFrames A and B using their respective id columns a_id and b_id<

Answer 1

Try this solution:

A_B = A.join(B,col('B.id') == col('A.id')).select([col('A.'+xx) for xx in A.columns]
      + [col('B.other1'),col('B.other2')])

The below lines in SELECT played the trick of selecting all columns from A and 2 columns from Table B.

[col('a.'+xx) for xx in a.columns] : all columns in a

[col('b.other1'),col('b.other2')] : some columns of b