While submit job with pyspark, how to access static files upload with --files argument?

问题

for example, i have a folder:

/
  - test.py
  - test.yml

and the job is submited to spark cluster with:

gcloud beta dataproc jobs submit pyspark --files=test.yml "test.py"

in the test.py, I want to access the static file I uploaded.

with open('test.yml') as test_file:
    logging.info(test_file.read())

but got the following exception:

IOError: [Errno 2] No such file or directory: 'test.yml'

How to access the file I uploaded?

回答1:

Files distributed using SparkContext.addFile (and --files) can be accessed via SparkFiles. It provides two methods:

• getRootDirectory() - returns root directory for distributed files
• get(filename) - returns absolute path to the file

I am not sure if there are any Dataproc specific limitations but something like this should work just fine:

from pyspark import SparkFiles

with open(SparkFiles.get('test.yml')) as test_file:
    logging.info(test_file.read())

回答2:

Yep, Shagun is right.

Basically when you submit a spark job to spark, it does not serialize the file you want processed over to each worker. You will have to do it yourself.

Typically, you will have to put the file in a shared file system like HDFS, S3 (amazon), or any other DFS that can be accessed by all the workers. As soon as you do that, and specify the file destination in your spark script, the spark job will be able to read and process as you wish.

However, having said this, copying the file into the same destination in ALL of you workers and master's file structure also work. Exp, you can create folders like /opt/spark-job/all-files/ in ALL spark nodes, rsync the file to all of them, and then you can use file in your spark script. But please do not do this. DFS or S3 are way better than this approach.

回答3:

Currently, as Dataproc is not in beta anymore, in order to direct access a file in the Cloud Storage from the PySpark code, submitting the job with --files parameter will do the work. SparkFiles is not required. For example:

gcloud dataproc jobs submit pyspark \
  --cluster *cluster name* --region *region name* \
  --files gs://<BUCKET NAME>/<FILE NAME> gs://<BUCKET NAME>/filename.py

While reading input from gcs via Spark API, it works with gcs connector.

来源：https://stackoverflow.com/questions/60231423/spark-submit-files-option-queries