Spark Streaming: Input Rate and File stream [0] has “Avg: 0.00 events/sec” always

丶灬走出姿态 提交于 2020-01-02 07:12:13

问题


I am running with spark 1.5.2, with below code. It is printing the count correctly at regular intervals, but in the spark streaming UI the Input Rate and File stream [0] has "Avg: 0.00 events/sec" always.

Note: Each file contains a single line containing a json string. I have also tried with each file containing multiple line, still same issue.

object main {
    def main(args: Array[String]) {
        val conf = new SparkConf().setAppName("test")
        val sc = new SparkContext(conf)
        val ssc = new StreamingContext(sc, Seconds(3))

        val dStream = ssc.textFileStream("srcDir")
        dStream.count().print()

        ssc.start()
        ssc.awaitTermination()
    }
}

来源:https://stackoverflow.com/questions/34195402/spark-streaming-input-rate-and-file-stream-0-has-avg-0-00-events-sec-alway

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!