hadoop2

I would like to create a file during my program. However, I don't want this file to be written on HDFS but on the datanode filesystem where the map operation is executed. I tried the following approach : public void map(Object key, Text value, Context context) throws IOException, InterruptedException { // do some hadoop stuff, like counting words String path = "newFile.txt"; try { File f = new File(path); f.createNewFile(); } catch (IOException e) { System.out.println("Message easy to look up in the logs."); System.err.println("Error easy to look up in the logs."); e.printStackTrace(); throw e

I have below kafka producer Api program and i am new to kafka itself. Below code fetch data from one of API and send message to kafka topic. package kafka_Demo; import java.util.Properties; import java.io.BufferedReader; import java.io.InputStream; import java.io.InputStreamReader; import org.apache.kafka.clients.producer.*; import java.net.URL; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; public class HttpBasicAuth { public static void main(String[] args) { try { Properties props = new Properties(); props.put("bootstrap

I am a bit confused with the output I get from Mapper. For example, when I run a simple wordcount program, with this input text: hello world Hadoop programming mapreduce wordcount lets see if this works 12345678 hello world mapreduce wordcount this is the output that I get: 12345678 1 Hadoop 1 hello 1 hello 1 if 1 lets 1 mapreduce 1 mapreduce 1 programming 1 see 1 this 1 wordcount 1 wordcount 1 works 1 world 1 world 1 As you can see, the output from mapper is already sorted. I did not run Reducer at all. But I find in a different project that the output from mapper is not sorted. So I am

Can anyone please tell me that If I am using java application to request some file upload/download operations to HDFS with Namenode HA setup, Where this request go first? I mean how would client know that which namenode is active? It would be great if you provide some workflow type diagram or something that explains request steps in detail(start to end). If hadoop cluster is configured with HA, then it will have namenode IDs in hdfs-site.xml like this : <property> <name>dfs.ha.namenodes.mycluster</name> <value>namenode1,namenode2</value> </property> Whichever NameNode is started first will

I have set the replication factor for my file as follows: hadoop fs -D dfs.replication=5 -copyFromLocal file.txt /user/xxxx When a NameNode restarts, it makes sure under-replicated blocks are replicated. Hence the replication info for the file is stored (possibly in nameNode ). How can I get that information? Try to use command hadoop fs -stat %r /path/to/file , it should print the replication factor. You can run following command to get replication factor, hadoop fs -ls /user/xxxx The second column in the output signify replication factor for the file and for the folder it shows - , as shown