pyspark Loading multiple partitioned files in a single load

Question

I am trying to load multiple files in a single load. They are all partitioned files When I tried it with 1 file it works, but when I listed down 24 files, it gives me this erro

Answer 1

As explained in the official documentation, to read multiple files, you should pass a list:

path – optional string or a list of string for file-system backed data sources.

So in your case:

(sqlContext.read
    .format('orc') 
    .options(basePath=basePath)
    .load(path=paths))

Argument unpacking (*) would makes sense only if load was defined with variadic arguments, form example:

def load(this, *paths):
    ...