Writing to HDFS from Java, getting “could only be replicated to 0 nodes instead of minReplication”

Question

I’ve downloaded and started up Cloudera\'s Hadoop Demo VM for CDH4 (running Hadoop 2.0.0). I’m trying to write a Java program that will run from my windows 7 machine (The same

Answer 1

Since I found many questions like this one in my search for having the exact same issue I thought I would share what finally worked for me. I found this forum post on Hortonworks: https://community.hortonworks.com/questions/16837/cannot-copy-from-local-machine-to-vm-datanode-via.html

The answer was truly understanding what calling new Configuration() means and setting the correct parameters as I needed them. In my case it was exactly the one mentioned in that post. So my working code looks like this.

try {
    Configuration config = new Configuration();
    config.set("dfs.client.use.datanode.hostname", "true");
    Path pdFile = new Path("stgicp-" + pd);
    FileSystem dFS = FileSystem.get(new URI("hdfs://" + HadoopProperties.HIVE_HOST + ":" + HadoopProperties.HDFS_DEFAULT_PORT), config, 
            HadoopProperties.HIVE_DEFAULT_USER);
    if (dFS.exists(pdFile)) {
        dFS.delete(pdFile, false);
    } 
    FSDataOutputStream outStream = dFS.create(pdFile);
    for (String sjWLR : processWLR.get(pd)) {
        outStream.writeBytes(sjWLR);
    }     
    outStream.flush();
    outStream.close();

    dFS.delete(pdFile, false);
    dFS.close();
} catch (IOException | URISyntaxException | InterruptedException e) {
    log.error("WLR file processing error: " + e.getMessage());
}