Use Spark to list all files in a Hadoop HDFS directory?

Question

I want to loop through all text files in a Hadoop dir and count all the occurrences of the word \"error\". Is there a way to do a hadoop fs -ls /users/ubuntu/ to li

Answer 1

You can use a wildcard:

val errorCount = sc.textFile("hdfs://some-directory/*")
                   .flatMap(_.split(" ")).filter(_ == "error").count