Error: AttributeError: 'DataFrame' object has no attribute '_jdf'

Question

I want to perform k-fold cross validation using pyspark to finetune the parameters and I\'m using pyspark.ml. I am getting Attribute Error.

AttributeError: \'DataFrame\'

Answer 1

Convert Panadas to Spark

from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

spark_dff = sqlContext.createDataFrame(panada_df)