Load multiple files in pig

Question

I am new to pig.

In pig, I want to load multiple files with timestamps at their names.

If I load files between 2012/02/12 to 2012/02/19, the following works<

Answer 1

The range you need can be loaded by using the following globs:

raw_data = load '/table/status/{2011/12/{29,3{0,1}},2012/01/0[1-4]}' using Loader()

Note, that Pig uses Hadoop's globbing so what you can use depends on the version of the underlying HDFS. This works under 0.20.

If a more complex range is needed then this way of directory enumeration might be a bit inflexible. Therefore consider using a shell script to gather the directories you need as suggested here