Dividing dataframes in pyspark

问题

Following up this question and dataframes, I am trying to convert this

Into this (I know it looks the same, but refer to the next code line to see the difference):

In pandas, I used the line code teste_2 = (value/value.groupby(level=0).sum()) and in pyspark I tried several solutions; the first one was:

 df_2 = (df/df.groupby(["age"]).sum())

However, I am getting the following error: TypeError: unsupported operand type(s) for /: 'DataFrame' and 'DataFrame'

The second one was:

df_2 = (df.filter(col('Siblings'))/gr.groupby(col('Age')).sum())

But it's still not working. Can anyone help me?

回答1:

Hope I've understood the question correctly. It seems you want to divide the count by the sum of count for each age group.

from pyspark.sql import functions as F, Window

df2 = df.groupBy('age', 'siblings').count().withColumn(
    'count',
    F.col('count') / F.sum('count').over(Window.partitionBy('age'))
)

df2.show()
+---+--------+-----+
|age|siblings|count|
+---+--------+-----+
| 15|       0|  1.0|
| 10|       3|  1.0|
| 14|       1|  1.0|
+---+--------+-----+

来源：https://stackoverflow.com/questions/65716510/dividing-dataframes-in-pyspark