pyspark mysql jdbc load An error occurred while calling o23.load No suitable driver

Question

I use docker image sequenceiq/spark on my Mac to study these spark examples, during the study process, I upgrade the spark inside that image to 1.6.1 according to this answe

Answer 1

I solved it using the following command

import findspark
findspark.add_packages('mysql:mysql-connector-java:8.0.11')

Answer 2

I ran into "java.sql.SQLException: No suitable driver" when I tried to have my script write to MySQL.

Here's what I did to fix that.

In script.py

df.write.jdbc(url="jdbc:mysql://localhost:3333/my_database"
                  "?user=my_user&password=my_password",
              table="my_table",
              mode="append",
              properties={"driver": 'com.mysql.jdbc.Driver'})

Then I ran spark-submit this way

SPARK_HOME=/usr/local/Cellar/apache-spark/1.6.1/libexec spark-submit --packages mysql:mysql-connector-java:5.1.39 ./script.py

Note that SPARK_HOME is specific to where spark is installed. For your environment this https://github.com/sequenceiq/docker-spark/blob/master/README.md might help.

In case all the above is confusing, try this:
In t.py replace

sqlContext.read.format("jdbc").option("url",url).option("dbtable","people").load()

with

sqlContext.read.format("jdbc").option("dbtable","people").option("driver", 'com.mysql.jdbc.Driver').load()

And run that with

spark-submit --packages mysql:mysql-connector-java:5.1.39 --master local[4] t.py