input-split

问题 Assume a client application that uses a FileSplit object in order to read the actual bytes from the corresponding file. To do so, an InputStream object has to be created from the FileSplit , via code like: FileSplit split = ... // The FileSplit reference FileSystem fs = ... // The HDFS reference FSDataInputStream fsin = fs.open(split.getPath()); long start = split.getStart()-1; // Byte before the first if (start >= 0) { fsin.seek(start); } The adjustment of the stream by -1 is present in some

问题 I'm writing a M/R job that processes large time-series-data files written in binary format that looks something like this (new lines here for readability, actual data is continuous, obviously): TIMESTAMP_1---------------------TIMESTAMP_1 TIMESTAMP_2**********TIMESTAMP_2 TIMESTAMP_3%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%TIMESTAMP_3 .. etc Where timestamp is simply a 8 byte struct, identifiable as such by the first 2 bytes. The actual data is bounded between duplicate value timestamps, as displayed

问题 If i have a compressed file of 1GB which is splittable and by default the block size and input split size is 128MB then there are 8 blocks created and 8 input split. When the compressed block is read by map reduce it is uncompressed and say after uncompression the size of the block becomes 200MB. But the input split for this assigned is of 128MB, so how is the rest of the 82MB processed. Is it processed by the next input split? Is the same input split size is increased? 回答1: Here is my