Replicate Spark Row N-times

Question

I want to duplicate a Row in a DataFrame, how can I do that?

For example, I have a DataFrame consisting of 1 Row, and I want to make a DataFrame with 100 identical R

Answer 1

You could pick out the single row, make a list with a hundred elements, populated with that row and convert it back into a dataframe.

import org.apache.spark.sql.DataFrame

val testDf = sc.parallelize(Seq(
    (1,2,3), (4,5,6)
)).toDF("one", "two", "three")

def replicateDf(n: Int, df: DataFrame) = sqlContext.createDataFrame(
    sc.parallelize(List.fill(n)(df.take(1)(0)).toSeq), 
    df.schema)

val replicatedDf = replicateDf(100, testDf)