add columns in dataframes dynamically with column names as elements in List

青春壹個敷衍的年華 提交于 2021-02-08 08:06:42

问题


I have List[N] like below

val check = List ("a","b","c","d")

where N can be any number of elements.

I have a dataframe with only column called "value". Based on the contents of value i need to create N columns with column names as elements in the list and column contents as substring(x,y)

I have tried all possible ways, like withColumn, selectExpr, nothing works. Please consider substring(X,Y) where X and Y as some numbers based on some metadata

Below are my different codes which I tried, but none worked,


val df = sqlContext.read.text("xxxxx")
val coder: (String => String) = (arg: String) => {
val param = "NULL"
if (arg.length() > Y )
arg.substring(X,Y)
else
val sqlfunc = udf(coder)
val check = List ("a","b","c","d")
for (name <- check){val testDF2 = df.withColumn(name, sqlfunc(df("value")))}

testDF2 has only last column d and other columns such as a,b,c are not added in table


var z:Array[String] = new Array[String](check.size)
var i=0
for ( x <- check ) {
if ( (i+1) == check.size) {
z(i) = s""""substring(a.value,X,Y) as $x""""
i = i+1}
else{
z(i) = s""""substring(a.value,X,Y) as $x","""
i = i+1}}
val zz = z.mkString(" ")
df.alias("a").selectExpr(s"$zz").show()

This throws error


Please help how to add columns in DF dynamically with column names as elements in List

I am expecting an Df like below

-----------------------------
Value| a | b | c | d | .... N
-----------------------------
|xxx|xxx|xxx|xxx|xxx|xxxxxx-                
|xxx|xxx|xxx|xxx|xxx|xxxxxx- 
|xxx|xxx|xxx|xxx|xxx|xxxxxx-
-----------------------------

回答1:


you can dynamically add columns from your list using for instance this answer by user6910411 to a similar question (see her/his full answer for more possibilities):

val newDF = check.foldLeft(<yourdf>)((df, name) => df.withColumn(name,<yourUDF>$"value"))



来源:https://stackoverflow.com/questions/42761328/add-columns-in-dataframes-dynamically-with-column-names-as-elements-in-list

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!