Read range of files in pySpark

问题

I need to read contiguous files in pySpark. The following works for me.

from pyspark.sql import SQLContext    
file = "events.parquet/exportDay=2015090[1-7]"
df = sqlContext.read.load(file)

How do I read files 8-14?

回答1:

Use curly braces.

file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"

Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).

回答2:

It uses shell globbing, I believe.

The post: How to read multiple text files into a single RDD?

Seems to suggest the below should work.

"events.parquet/exportDay=2015090[89],events.parquet/exportDay=2015091[0-4]"

来源：https://stackoverflow.com/questions/37732766/read-range-of-files-in-pyspark