While submit job with pyspark, how to access static files upload with --files argument?

Question

for example, i have a folder:

/
  - test.py
  - test.yml

and the job is submited to spark cluster with:

gcloud beta dataproc jobs

Answer 1

Files distributed using SparkContext.addFile (and --files) can be accessed via SparkFiles. It provides two methods:

• getRootDirectory() - returns root directory for distributed files
• get(filename) - returns absolute path to the file

I am not sure if there are any Dataproc specific limitations but something like this should work just fine:

from pyspark import SparkFiles

with open(SparkFiles.get('test.yml')) as test_file:
    logging.info(test_file.read())

Answer 2

Currently, as Dataproc is not in beta anymore, in order to direct access a file in the Cloud Storage from the PySpark code, submitting the job with --files parameter will do the work. SparkFiles is not required. For example:

gcloud dataproc jobs submit pyspark \
  --cluster *cluster name* --region *region name* \
  --files gs://<BUCKET NAME>/<FILE NAME> gs://<BUCKET NAME>/filename.py

While reading input from gcs via Spark API, it works with gcs connector.

Answer 3

Yep, Shagun is right.

Basically when you submit a spark job to spark, it does not serialize the file you want processed over to each worker. You will have to do it yourself.

Typically, you will have to put the file in a shared file system like HDFS, S3 (amazon), or any other DFS that can be accessed by all the workers. As soon as you do that, and specify the file destination in your spark script, the spark job will be able to read and process as you wish.

However, having said this, copying the file into the same destination in ALL of you workers and master's file structure also work. Exp, you can create folders like /opt/spark-job/all-files/ in ALL spark nodes, rsync the file to all of them, and then you can use file in your spark script. But please do not do this. DFS or S3 are way better than this approach.