Spark SQL Row_number() PartitionBy Sort Desc

Question

I\'ve successfully create a row_number() partitionBy by in Spark using Window, but would like to sort this by descending, instead of the default ascend

Answer 1

desc should be applied on a column not a window definition. You can use either a method on a column:

from pyspark.sql.functions import col, row_number
from pyspark.sql.window import Window

F.row_number().over(
    Window.partitionBy("driver").orderBy(col("unit_count").desc())
)

or a standalone function:

from pyspark.sql.functions import desc
from pyspark.sql.window import Window

F.row_number().over(
    Window.partitionBy("driver").orderBy(desc("unit_count"))
)