AttributeError: 'NoneType' object has no attribute 'setCallSite'

Question

In PySpark, I want to calculate the correlation between two dataframe vectors, using the following code (I do not have any problem in importing pyspark or createDataFrame):

Answer 1

I got the same error not only with Correlation.corr(...) dataframe, but with ldaModel.describeTopics() as well.

Most probably it is the SPARK bug.

They forget to initialise DataFrame::_sc._jsc member when created resulting dataframe.

Each dataframe has normally this member initialised with proper JavaObject.