Spark losing println() on stdout

假如想象 提交于 2019-11-26 06:44:57

问题


I have the following code:

val blueCount = sc.accumulator[Long](0)
val output = input.map { data =>
  for (value <- data.getValues()) {
    if (record.getEnum() == DataEnum.BLUE) {
      blueCount += 1
      println(\"Enum = BLUE : \" + value.toString()
    }
  }
  data
}.persist(StorageLevel.MEMORY_ONLY_SER)

output.saveAsTextFile(\"myOutput\")

Then the blueCount is not zero, but I got no println() output! Am I missing anything here? Thanks!


回答1:


This is a conceptual question...

Imagine You have a big cluster, composed of many workers let's say n workers and those workers store a partition of an RDD or DataFrame, imagine You start a map task across that data, and inside that map you have a print statement, first of all:

  • Where will that data be printed out?
  • What node has priority and what partition?
  • If all nodes are running in parallel, who will be printed first?
  • How will be this print queue created?

Those are too many questions, thus the designers/maintainers of apache-spark decided logically to drop any support to print statements inside any map-reduce operation (this include accumulators and even broadcast variables).

This also makes sense because Spark is a language designed for very large datasets. While printing can be useful for testing and debugging, you wouldn't want to print every line of a DataFrame or RDD because they are built to have millions or billions of rows! So why deal with these complicated questions when you wouldn't even want to print in the first place?

In order to prove this you can run this scala code for example:

// Let's create a simple RDD
val rdd = sc.parallelize(1 to 10000)

def printStuff(x:Int):Int = {
  println(x)
  x + 1
}

// It doesn't print anything! because of a logic design limitation!
rdd.map(printStuff)

// But you can print the RDD by doing the following:
rdd.take(10).foreach(println)



回答2:


I was able to work it around by making a utility function:

object PrintUtiltity {
    def print(data:String) = {
      println(data)
    }
}


来源:https://stackoverflow.com/questions/33225994/spark-losing-println-on-stdout

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!