问题
I am new to pyspark. I am trying to understand how to access parquet file with multiple level of nested struct and array's. I need to replace some value in a data-frame (with nested schema) with null, I have seen this solution it works fine with structs but it not sure how this works with arrays.
My schema is something like this
|-- unitOfMeasure: struct
| |-- raw: struct
| | |-- id: string
| | |-- codingSystemId: string
| | |-- display: string
| |-- standard: struct
| | |-- id: string
| | |-- codingSystemId: string
|-- Id: string
|-- actions: array
| |-- element: struct
| | |-- action: string
| | |-- actionDate: string
| | |-- actor: struct
| | | |-- actorId: string
| | | |-- aliases: array
| | | | |-- element: struct
| | | | | |-- value: string
| | | | | |-- type: string
| | | | | |-- assigningAuthority: string
| | | |-- fullName: string
What I wanted to do is replace unitOfMeasure.raw.id to null
and actions.element.action with null
and actions.element.actor.aliases.element.value with null keep the rest of my data frame untouched.
Is there any way that I can achieve this?
回答1:
For array columns, it's a bit complicated compared to struct fields. One option is to explode the array into new column so that you could access and update the nested structs. After the update, you'll have to reconstruct the initial array column.
But I'd prefer using higher-order function transform which is introduced for Spark >=2.4 Here is an example:
Input DF:
|-- actions: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- action: string (nullable = true)
| | |-- actionDate: string (nullable = true)
| | |-- actor: struct (nullable = true)
| | | |-- actorId: long (nullable = true)
| | | |-- aliases: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- assigningAuthority: string (nullable = true)
| | | | | |-- type: string (nullable = true)
| | | | | |-- value: string (nullable = true)
| | | |-- fullName: string (nullable = true)
+--------------------------------------------------------------+
|actions |
+--------------------------------------------------------------+
|[[action_name1, 2019-12-08, [2, [[aa, t1, v1]], full_name1]]] |
|[[action_name2, 2019-12-09, [3, [[aaa, t2, v2]], full_name2]]]|
+--------------------------------------------------------------+
We pass a lambda function to transfrom which select all the struct fields and replace actions.action and actions.actor.aliases.value by null.
transform_expr = """transform (actions, x ->
struct(null as action,
x.actionDate as actionDate,
struct(x.actor.actorId as actorId,
transform(x.actor.aliases, y ->
struct(null as value,
y.type as type,
y.assigningAuthority as assigningAuthority)
) as aliases,
x.actor.fullName as fullName
) as actor
))"""
df.withColumn("actions", expr(transform_expr)).show(truncate=False)
Output DF:
+------------------------------------------------+
|actions |
+------------------------------------------------+
|[[, 2019-12-08, [2, [[, t1, aa]], full_name1]]] |
|[[, 2019-12-09, [3, [[, t2, aaa]], full_name2]]]|
+------------------------------------------------+
来源：https://stackoverflow.com/questions/59237945/replace-value-in-deep-nested-schema-spark-dataframe