Pyspark filter dataframe by columns of another dataframe

Question

Not sure why I\'m having a difficult time with this, it seems so simple considering it\'s fairly easy to do in R or pandas. I wanted to avoid using pandas though since I\'m

Answer 1

Left anti join is what you're looking for:

df1.join(df2, ["userid", "group"], "leftanti")

but the same thing can be done with left outer join:

(df1
    .join(df2, ["userid", "group"], "leftouter")
    .where(df2["pick"].isNull())
    .drop(df2["pick"]))