google-cloud-dataproc

I am using maven shade plugin to build Uber jar for submitting it as a job to google dataproc cluster. Google have installed Apache Spark 2.0.2 Apache Hadoop 2.7.3 on their cluster. Apache spark 2.0.2 uses 14.0.1 of com.google.guava and apache hadoop 2.7.3 uses 11.0.2, these both should be in the classpath already. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <!-- <artifactSet> <includes> <include>com.google.guava:guava

I was encountering stackoverflowerrors when I was iteratively adding over 500 columns to my pyspark dataframe. So, I included checkpoints. The checkpoints did not help. So, I created the following toy application to test if my checkpoints were working correctly. All I do in this example is iteratively create columns by copying the original column over and over again. I persist, checkpoint and count every 10 iterations. I notice that my dataframe.rdd.isCheckpointed() always returns False. I can verify that the checkpoint folders are indeed being created and populated on disk. I am running on

After a dataproc cluster is created, many jobs are submitted automatically to ResourceManager by user dr.who. This is starving the resources of the cluster and eventually overwhelms the cluster so. There is little to no information in the logs. Is anyone else experiencing this issue in dataproc? Without knowing more, here is what I suspect is going on. It sounds like your cluster has been compromised Your firewall (network) rules are likely open, allowing any traffic into the cluster Someone has discovered your cluster is open to the public internet and is taking advantage of it I recommend

When running a PySpark job on the dataproc server like this gcloud --project <project_name> dataproc jobs submit pyspark --cluster <cluster_name> <python_script> my print statements don't show up in my terminal. Is there any way to output data onto the terminal in PySpark when running jobs on the cloud? Edit : I would like to print/log info from within my transformation. For example: def print_funct(l): print(l) return l rddData.map(lambda l: print_funct(l)).collect() Should print every line of data in the RDD rddData . Doing some digging, I found this answer for logging , however, testing it

Looking for expertise to guide me on issue below. Background: I'm trying to get going with a basic PySpark script inspired on this example As deploy infrastructure I use a Google Cloud Dataproc Cluster. Cornerstone in my code is the function "recommendProductsForUsers" documented here which gives me back the top X products for all users in the model Issue I incur The ALS.Train script runs smoothly and scales well on GCP (Easily >1mn customers). However, applying the predictions: i.e. using funcitons 'PredictAll' or 'recommendProductsForUsers', does not scale at all. My script runs smooth for a

I'm running a .jar file that contains all dependencies that I need packaged in it. One of this dependencies is com.google.common.util.concurrent.RateLimiter and already checked it's class file is in this .jar file. Unfortunately when I hit the command spark-submit on the master node of my google's dataproc-cluster instance I'm getting this error: Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch; at com.google.common.util.concurrent.RateLimiter$SleepingStopwatch$1.<init>(RateLimiter.java:417) at com.google