Spark joins not working in Spark > 2.4 when the join column name is same

Question

We recently upgraded Spark from v.2.1 to v.2.4

The code which was working previously is as below:

final_df = spark.sql(&quo