google-cloud-dataproc

When starting Jupyter Notebook on Google Dataproc, importing modules fails. I have tried to install the modules using different commands. Some examples: import os os.sytem("sudo apt-get install python-numpy") os.system("sudo pip install numpy") #after having installed pip os.system("sudo pip install python-numpy") #after having installed pip import numpy None of the above examples work and return an import error: enter image description here When using command line I am able to install modules, but still the import error remains. I guess I am installing modules in a wrong location. Any

I have a data table loaded in bigquery, and I want to import it in my spark cluster via a pyspark .py file. I saw in Dataproc + BigQuery examples - any available? that there was a way to load a bigquery table in the spark cluster with scala, but is there a way to do it in a pyspark script? James This comes from @MattJ in this question . Here's an example to connect to BigQuery in Spark and perform a word count. import json import pyspark sc = pyspark.SparkContext() hadoopConf=sc._jsc.hadoopConfiguration() hadoopConf.get("fs.gs.system.bucket") conf = {"mapred.bq.project.id": "<project_id>",

is there a way of pausing a Dataproc cluster so I don't get billed when I am not actively running spark-shell or spark-submit jobs ? The cluster management instructions at this link: https://cloud.google.com/sdk/gcloud/reference/beta/dataproc/clusters/ only show how to destroy a cluster but I have installed spark cassandra connector API for example. Is my only alternative to just creating an image that I'll need to install every time ? In general, the best thing to do is to distill out the steps you used to customize your cluster into some setup scripts, and then use Dataproc's initialization

i'm getting this error while running pyspark job on dataproc. What could be the reason ? This is the stack trace of error. File "/usr/lib/python2.7/pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 553, in save_reduce File "/usr/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python2.7/pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems save(v) File "/usr/lib