Column data to nested json object in Spark structured streaming

Question

In our application we obtain the field values as columns using Spark sql. Im\' trying to figure out how to put the columns values to nested json object and push to Elasticsearch

Answer 1

Adding all columns to a top level struct should give the expected output. In Scala:

import org.apache.spark.sql.functions._

data.select(struct(data.columns:_*).as("txn_summary"))

In Java I would suspect it it would be:

import org.apache.spark.sql.functions.struct;

data.select(struct(data.columns()).as("txn_summary"));