Hadoop Pseudo-Distributed Operation error: Protocol message tag had invalid wire type

Question

I am setting up a Hadoop 2.6.0 Single Node Cluster. I follow the hadoop-common/SingleCluster documentation. I work on Ubuntu 14.04. So far I have m

Answer 1

i got this error once when i was uploading files to hdfs from java code ,The issue was that i was using hadoop 1 jar to connect to hadoop 2 installation,not sure whats the problem in your case but if you ever configured hadoop 1 eariler then something must be messing with it

Answer 2

This is a set of steps I followed on Ubuntu when facing exactly the same problem but with 2.7.1, the steps shouldn't differ much for previous and future version (I'd believe).

1) Format of my /etc/hosts folder:

    127.0.0.1    localhost   <computer-name>
    # 127.0.1.1    <computer-name>
    <ip-address>    <computer-name>

    # Rest of file with no changes

2) *.xml configuration files (displaying contents inside <configuration> tag):

• For core-site.xml:

      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost/</value>
      </property>
      <!-- set value to a directory you want with an absolute path -->
      <property>
          <name>hadoop.tmp.dir</name>
          <value>"set/a/directory/on/your/machine/"</value>
          <description>A base for other temporary directories</description>
      </property>
  

• For hdfs-site.xml:

      <property>
          <name>dfs.replication</name>
          <value>1</value>
      </property>
  

• For yarn-site.xml:

      <property>
          <name>yarn.recourcemanager.hostname</name>
          <value>localhost</value>
      </property>
  
      <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
      </property>
  

• For mapred-site.xml:

      <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
      </property>
  

3) Verify $HADOOP_CONF_DIR:

This is a good opportunity to verify that you are indeed using this configuration. In the folder where your .xml files reside, view contents of script hadoop_env.sh and make sure $HADOOP_CONF_DIR is pointing at the right directory.

4) Check your PORTS:

NameNode binds ports 50070 and 8020 on my standard distribution and DataNode binds ports 50010, 50020, 50075 and 43758. Run sudo lsof -i to be certain no other services are using them for some reason.

5) Format if necessary:

At this point, if you have changed the value hadoop.tmp.dir you should reformat the NameNode by hdfs namenode -format. If not remove the temporary files already present in the tmp directory you are using (default /tmp/):

6) Start Nodes and Yarn:

In /sbin/ start the name and data node by using the start-dfs.sh script and yarn with start-yarn.sh and evaluate the output of jps:

    ./start-dfs.sh   
    ./start-yarn.sh

---

At this point if NameNode, DataNode, NodeManager and ResourceManager are all running you should be set to go!

If any of these hasn't started, share the log output for us to re-evaluate.

Answer 3

Do these changes in /etc/hosts:

1. Change:

127.0.0.1    localhost

to

127.0.0.1    localhost    marta-komputer

2. Delete:

127.0.0.1    marta-komputer

3. Add:

your-system-ip    marta-komputer

To find your system IP, type this in terminal

ifconfig

(find your IP address here) or type this:

ifdata -pa eth0

Your final /etc/hosts file should look like:

127.0.0.1       localhost       marta-komputer
your-system-ip       marta-komputer

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Change hdfs-site.xml:

1. Change:

hdfs://localhost:9000

to

hdfs://marta-komputer:9000

Now, stop and start hadoop processes.

Your jps command should list these processes:

Namenode
Datanode
TaskTracker
SecondaryNameNode

If it does not list all these processes, check respective logs for errors.

UPDATE:

1. Follow this tutorial here

2. If the problem persists, it might be due to permission issue.

UPDATE II:

1. Create a directory and change permissions for namenode and datanode:

sudo mkdir -p /usr/local/hdfs/namenode

sudo mkdir -p /usr/local/hdfs/datanode

sudo chown -R hduser:hadoop /usr/local/hdfs/namenode

sudo chown -R hduser:hadoop /usr/local/hdfs/datanode

2. Add these properties in hdfs-site.xml:

dfs.datanode.data.dir with value /usr/local/hdfs/datanode

dfs.namenode.data.dir with value /usr/local/hdfs/namenode

3. Stop and start hadoop processes.

Answer 4

remove 127.0.0.1 localhost from /etc/hosts and change your core-site.xml like follow:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://marta-komputer:9000</value>
    </property>
</configuration>

and you can ignore the WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... warning