Spark dataframe: Pivot and Group based on columns

Question

I have input dataframe as below with id, app, and customer

Input dataframe

+--------------------+-----+---------+
|                          


        

Answer 1

Using CONCAT_WS we can explode array and can remove the square brackets.

df.groupBy("id").pivot("app").agg(concat_ws(",",collect_list("customer")))

Answer 2

You can use collect_list if you can bear with an empty List at cells where it should be zero:

df.groupBy("id").pivot("app").agg(collect_list("customer")).show
+---+--------+----+--------+
| id|      bc|  fe|      fw|
+---+--------+----+--------+
|id3|[TR, WM]|  []|      []|
|id1|      []|[WM]|[CS, WM]|
|id2|      []|  []|    [CS]|
+---+--------+----+--------+