Spark 3 BHJ of CSV DataFrame throws java.util.NoSuchElementException: key not found: numPartitions

Question

While upgrading a Spark 2.4.6 codebase to 3.0.1, I encouter a seemingly weird exception when joining a very small DataFrame with a union of two larger ones: