Query A Nested Array in Parquet Records

Question

I am trying different ways to query a record within a array of records and display complete Row as output.

I dont know which nested Object has String \"pg\"

Answer 1

It seems that you can use

org.apache.spark.sql.functions.explode(e: Column): Column

for example in my project(in java), i have nested json like this:

{
    "error": [],
    "trajet": [
        {
            "something": "value"
        }
    ],
    "infos": [
        {
            "something": "value"
        }
    ],
    "timeseries": [
        {
            "something_0": "value_0",
            "something_1": "value_1",
            ...
            "something_n": "value_n"
        }
    ]
}

and i wanted to analyse datas in "timeseries", so i did:

DataFrame ts = jsonDF.select(org.apache.spark.sql.functions.explode(jsonDF.col("timeseries")).as("t"))
                     .select("t.something_0",
                             "t.something_1",
                             ...
                             "t.something_n");

I'm new to spark too. Hope this could give you a hint.

Answer 2

The problem was solved by

I found a way through Explode.

val results = sqlc.sql("SELECT firstname, child.name, FROM parent LATERAL VIEW explode(children) childTable AS child