pyspark EOFError after calling map

Question

I am new to spark & pyspark.

I am reading a small csv file (~40k) into a dataframe.

from pyspark.sql import functions as F
df = sqlContext.read.forma         


        

Answer 1

Can you please try to do map after converting dataframe into rdd. You are applying map function on a dataframe and then again creating a dataframe from that.Syntax would be like

df.rdd.map().toDF()

Please let me know if it works. Thanks.