saving prediction from pyspark into HDFS is very slow on yarn (cluster or client)

Question

df = rf_model.transform(df).select("id", "probability", "prediction")

df = df.withColumn("prediction", df["prediction"