optimizing shuffle spark outer join dataset

Question

I am using Spark 2.1 with DataFrames API to do :

import org.apache.spark.sql.Encoders
import java.security.MessageDigest
import org.apache.spark.sql.functions