Spark dataframes convert nested JSON to seperate columns

Question

I\'ve a stream of JSONs with following structure that gets converted to dataframe

{
  \"a\": 3936,
  \"b\": 123,
  \"c\": \"34\",
  \"attributes\": {
    \"         


        

Answer 1

• If you want columns named from a to f:

  df.select("a", "b", "c", "attributes.d", "attributes.e", "attributes.f")
  

• If you want columns named with attributes. prefix:

  df.select($"a", $"b", $"c", $"attributes.d" as "attributes.d", $"attributes.e" as "attributes.e", $"attributes.f" as "attributes.f")
  

• If names of your columns are supplied from an external source (e.g. configuration):

  val colNames: Seq("a", "b", "c", "attributes.d", "attributes.e", "attributes.f")
  
  df.select(colNames.head, colNames.tail: _*).toDF(colNames:_*)