melt

问题 This question already has answers here : Reshaping multiple sets of measurement columns (wide format) into single columns (long format) (7 answers) Closed 2 years ago . I need to reshape my wide table into long format but keeping multiple fields for each record, for example: dw <- read.table(header=T, text=\' sbj f1.avg f1.sd f2.avg f2.sd blabla A 10 6 50 10 bA B 12 5 70 11 bB C 20 7 20 8 bC D 22 8 22 9 bD \') # Now I want to melt this table, keeping both AVG and SD as separate fields for

问题 Is there an equivalent of Pandas Melt Function in Apache Spark in PySpark or at least in Scala? I was running a sample dataset till now in python and now I want to use Spark for the entire dataset. Thanks in advance. 回答1: There is no built-in function (if you work with SQL and Hive support enabled you can use stack function, but it is not exposed in Spark and has no native implementation) but it is trivial to roll your own. Required imports: from pyspark.sql.functions import array, col,