Spark solution for below problem (Transform , Pivot, CrossJoin)

Question

This is the input dataframe:

df1_input = spark.createDataFrame([ \\
    ("P1","A","B","C"), \\
    ("P1",&qu