问题
We are working with Spark Standalone Cluster with 8 Cores and 32GB Ram, with 3 nodes cluster with same configuration.
Some times streaming batch completed in less than 1sec. some times it takes more than 10 secs at that time below log will appears in console.
2016-03-29 11:35:25,044 INFO TaskSchedulerImpl:59 - Removed TaskSet 18.0, whose tasks have all completed, from pool
2016-03-29 11:35:25,044 INFO DAGScheduler:59 - Job 18 finished: foreachRDD at EventProcessor.java:87, took 1.128755 s
2016-03-29 11:35:31,471 INFO JobScheduler:59 - Added jobs for time 1459231530000 ms
2016-03-29 11:35:35,004 INFO JobScheduler:59 - Added jobs for time 1459231535000 ms
2016-03-29 11:35:40,004 INFO JobScheduler:59 - Added jobs for time 1459231540000 ms
2016-03-29 11:35:45,136 INFO JobScheduler:59 - Added jobs for time 1459231545000 ms
2016-03-29 11:35:50,011 INFO JobScheduler:59 - Added jobs for time 1459231550000 ms
2016-03-29 11:35:55,004 INFO JobScheduler:59 - Added jobs for time 1459231555000 ms
2016-03-29 11:36:00,014 INFO JobScheduler:59 - Added jobs for time 1459231560000 ms
2016-03-29 11:36:05,003 INFO JobScheduler:59 - Added jobs for time 1459231565000 ms
2016-03-29 11:36:10,087 INFO JobScheduler:59 - Added jobs for time 1459231570000 ms
2016-03-29 11:36:15,004 INFO JobScheduler:59 - Added jobs for time 1459231575000 ms
2016-03-29 11:36:20,004 INFO JobScheduler:59 - Added jobs for time 1459231580000 ms
2016-03-29 11:36:25,139 INFO JobScheduler:59 - Added jobs for time 1459231585000 ms
Can you please help, how to solve this problem.
回答1:
Change the spark-submit master from local to local[2]
spark-submit --master local[2] --class YOURPROGRAM YOUR.jar
Or set
new SparkConf().setAppName("SparkStreamingExample").setMaster("local[2]")
If you still facing the same problem after changing the number to 2, maybe you should just change it to a bigger number.
Reference: http://spark.apache.org/docs/latest/streaming-programming-guide.html
When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using an input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data. Hence, when running locally, always use “local[n]” as the master URL, where n > the number of receivers to run (see Spark Properties for information on how to set the master).
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise, the system will receive data, but not be able to process them.
Credit to bit1129: http://bit1129.iteye.com/blog/2174751
回答2:
I solved this problem by setting master from local to local[2]. Following related quote is from spark streaming doc:
But note that a Spark worker/executor is a long-running task, hence it occupies one of the cores allocated to the Spark Streaming application. Therefore, it is important to remember that a Spark Streaming application needs to be allocated enough cores (or threads, if running locally) to process the received data, as well as to run the receiver(s).
回答3:
It's not a problem indeed, these INFOs are just log messages which you can avoid by changing log level from INFO to WARN or ERROR in conf/log4j.properties.
Spark Streaming would buffer your input data into small batches and submit the batch of input for execution periodically, therefore not a problem here.
来源：https://stackoverflow.com/questions/36281665/continuously-info-jobscheduler59-added-jobs-for-time-ms-in-my-spark-stand