Spark DataFrame exploding a map with the key as a member

Question

I\'ve found a map exploding example at databrick\'s blog:

// input
{
  \"a\": {
    \"b\": 1,
    \"c\": 2
  }
}

Python: events.select(explode(\"a\").alias(         


        

Answer 1

Although I don't know whether its possible to explode the map with one single explode, there is a way to it with a UDF. The trick is to use Row#schema.fields(i).name to get the name of the "key"

def mapStructs = udf((r: Row) => {
  r.schema.fields.map(f => (
    f.name,
    r.getAs[Row](f.name).getAs[Long]("d"),
    r.getAs[Row](f.name).getAs[Long]("e"))
  )
})

df
  .withColumn("udfResult", explode(mapStructs($"a")))
  .withColumn("x", $"udfResult._1")
  .withColumn("d", $"udfResult._2")
  .withColumn("e", $"udfResult._3")
  .drop($"udfResult")
  .drop($"a")
  .show

gives

+---+---+---+---+
| id|  x|  d|  e|
+---+---+---+---+
|  0|  b|  1|  2|
|  0|  c|  3|  4|
+---+---+---+---+