Run a Local file system directory as input of a Mapper in cluster

后端 未结 7 989
刺人心
刺人心 2021-01-12 13:22

I gave an input to the mapper from a local filesystem.It is running successfully from eclipse,But not running from the cluster as it is unable to find the local input path s

相关标签:
7条回答
  • 2021-01-12 14:09

    This is a very old question. Recently faced the same issue. I am not aware of how correct this solution is it worked for me though. Please bring to notice if there are any drawbacks of this.Here's what I did.

    Reading a solution from the mail-archives, I realised if i modify fs.default.name from hdfs://localhost:8020/ to file:/// it can access the local file system. However, I didnt want this for all my mapreduce jobs. So I made a copy of core-site.xml in a local system folder (same as the one from where I would submit my MR jar to hadoop jar).

    and in my Driver class for MR I added,

    Configuration conf = new Configuration();
    conf.addResource(new Path("/my/local/system/path/to/core-site.xml"));
    conf.addResource(new Path("/usr/lib/hadoop-0.20-mapreduce/conf/hdfs-site.xml"));
    

    The MR takes input from local system and writes the output to hdfs:

    0 讨论(0)
提交回复
热议问题