How to name file when saveAsTextFile in spark?

Question

When saving as a textfile in spark version 1.5.1 I use: rdd.saveAsTextFile(\'\').

But if I want to find the file in that direcotry, how d

Answer 1

The correct answer to this question is that saveAsTextFile does not allow you to name the actual file.

The reason for this is that the data is partitioned and within the path given as a parameter to the call to saveAsTextFile(...), it will treat that as a directory and then write one file per partition.

You can call rdd.coalesce(1).saveAsTextFile('/some/path/somewhere') and it will create /some/path/somewhere/part-0000.txt.

If you need more control than this, you will need to do an actual file operation on your end after you do a rdd.collect().

Notice, this will pull all data into one executor so you may run into memory issues. That's the risk you take.