问题
I am running with spark 1.5.2, with below code. It is printing the count correctly at regular intervals, but in the spark streaming UI the Input Rate and File stream [0] has "Avg: 0.00 events/sec" always.
Note: Each file contains a single line containing a json string. I have also tried with each file containing multiple line, still same issue.
object main {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("test")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(3))
val dStream = ssc.textFileStream("srcDir")
dStream.count().print()
ssc.start()
ssc.awaitTermination()
}
}
来源:https://stackoverflow.com/questions/34195402/spark-streaming-input-rate-and-file-stream-0-has-avg-0-00-events-sec-alway