py4j

I have some third party Database client libraries in Java. I want to access them through java_gateway.py E.g: to make the client class (not a jdbc driver!) available to the python client via the java gateway: java_import(gateway.jvm, "org.mydatabase.MyDBClient") It is not clear where to add the third party libraries to the jvm classpath. I tried to add to compute-classpath.sh but that did nto seem to work: I get Py4jError: Trying to call a package Also, when comparing to Hive: the hive jar files are NOT loaded via compute-classpath.sh so that makes me suspicious. There seems to be some other

This is the snippet: from pyspark import SparkContext from pyspark.sql.session import SparkSession sc = SparkContext() spark = SparkSession(sc) d = spark.read.format("csv").option("header", True).option("inferSchema", True).load('file.csv') d.show() After this runs into the error: An error occurred while calling o163.showString. Trace: py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Integer, class java.lang.Boolean]) does not exist All the other methods work well. Tried researching alot but in vain. Any lead will be highly appreciated This is an indicator of a

I installed Spark, ran the sbt assembly, and can open bin/pyspark with no problem. However, I am running into problems loading the pyspark module into ipython. I'm getting the following error: In [1]: import pyspark --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-1-c15ae3402d12> in <module>() ----> 1 import pyspark /usr/local/spark/python/pyspark/__init__.py in <module>() 61 62 from pyspark.conf import SparkConf ---> 63 from pyspark.context import SparkContext 64 from pyspark.sql import SQLContext 65 from