问题
I got an array column with 512 double elements, and want to get the average. Take an array column with length=3 as example:
val x = Seq("2 4 6", "0 0 0").toDF("value").withColumn("value", split($"value", " "))
x.printSchema()
x.show()
root
|-- value: array (nullable = true)
| |-- element: string (containsNull = true)
+---------+
| value|
+---------+
|[2, 4, 6]|
|[0, 0, 0]|
+---------+
The following result is desired:
x.select(..... as "avg_value").show()
------------
|avg_value |
------------
|[1,2,3] |
------------
回答1:
Consider each array element as column and calculate average then construct array with those columns:
val array_size = 3
val avgAgg = for (i <- 0 to array_size -1) yield avg($"value".getItem(i))
df.select(array(avgAgg: _*).alias("avg_value")).show(false)
Gives:
+---------------+
|avg_value |
+---------------+
|[1.0, 2.0, 3.0]|
+---------------+
回答2:
This should do the trick for a constant sized array:
from pyspark.sql.functions import col, avg, array
df = spark.createDataFrame([ list([[x,x+1,x+2]]) for x in range(3)], ['value'])
num_array_elements = len(df.select("value").first()[0])
df.agg(array(*[avg(col("value")[i]) for i in range(num_array_elements)]).alias("avgValuesPerElement")).show()
returns:
+------------------+
|avgValesPerElement|
+------------------+
| [1.0, 2.0, 3.0]|
+------------------+
Thought I read pyspark. Leaving for pysparkers.
来源：https://stackoverflow.com/questions/59532225/how-to-obtain-the-average-of-an-array-type-column-in-scala-spark-over-all-row-en