Spark doesnt print outputs on the console within the map function

Question

I have a simple Spark application running on cluster mode.

val funcGSSNFilterHeader = (x: String) => {
    println(!x.contains(\"servedMSISDN\")   
    !         


        

Answer 1

Two possibilities: Your logs are on worker nodes, so you must check worker logs for these log messages. As suggested before, you can run your application in local mode to check logs on your machine. By the way, it's better to use i.e. SLF4j than just println, but I assume it's only for learning :)

In snippet there is no ssc.start() and ssc.awaitTermination(). Did you run these commands? If not, foreachRDD will not be executed any time. If the example is ok, please add these line at the end of script and try again, but please check worker nodes logs :)