spark ssc.textFileStream is not streamining any files from directory

Question

I am trying to execute below code using eclipse (with maven conf) with 2 worker and each have 2 core or also tried with spark-submit.

public class StreamingW         


        

Answer 1

JavaDoc suggests function only streams new files.

Ref: https://spark.apache.org/docs/1.0.1/api/java/org/apache/spark/streaming/api/java/JavaStreamingContext.html#textFileStream(java.lang.String)

Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat). Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.

Answer 2

You have to take in count that Spark Streaming will only read the new files in the directory, no the updated ones (once they are in the directory) and also they all must have the same format.

Source

Answer 3

Try it with another directory and then copy these files to that directory, while the job is running.

Answer 4

textFileStream can only monitor a folder when the files in the folder are being added or updated.

If you just want to read files, you can rather use SparkContext.textFile.

Answer 5

had the same problem. Here is my code:

lines = jssc.textFileStream("file:///Users/projects/spark/test/data');

the TextFileSTream is very sensitive; what i ended up doing was:

1. Run Spark program
2. touch datafile
3. mv datafile datafile2
4. mv datafile2  /Users/projects/spark/test/data

and that did it.

Answer 6

I think you need to add the scheme, i.e. file:// or hdfs:// in front of your path.

---

Undoing the edit to my comment because: It is in fact file:// and hdfs:// which needs to be added "in front of" the path, so the total path becomes file:///tmp/file.txt or hdfs:///user/data. If there is no NameNode set in the configuration, the latter needs to be hdfs://host:port/user/data.