PySpark in iPython notebook raises Py4JJavaError when using count() and first()

Question

I am using PySpark(v.2.1.0) in iPython notebook (python v.3.6) over virtualenv in my Mac(Sierra 10.12.3 Beta).

1.I launched iPython notebook by shooting this in Ter

Answer 1

Yeah I had the same problem long time ago in Pyspark in Anaconda I tried several ways to rectify this finally I found on my own by installing Java for anaconda separately afterwards there is no Py4jerror.

https://anaconda.org/cyclus/java-jdk

Answer 2

If you are using Anaconda, try to install java-jdk for Anaconda:

conda install -c cyclus java-jdk

Answer 3

Pyspark 2.1.0 is not compatible with python 3.6, see https://issues.apache.org/jira/browse/SPARK-19019.

You need to use earlier python version or you can try building master or 2.1 branch from github and it should work.