hadoop2

问题 I'm trying to write unit tests using miniDFSCluster and it's throwing the error below java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z any pointers to resolve this issue? 回答1: With errors like this, I use three steps Find out what it is looking for In this case, *org.apache.hadoop.io.nativeio.NativeIO$Windows.access0* Find out what jar/lib it is in. I don't use the Windows version, but I believe it is in hadoop.dll - you'll have to

问题 I am migrating my application from hadoop 1.0.3 to hadoop 2.2.0 and maven build had hadoop-core marked as dependency. Since hadoop-core is not present for hadoop 2.2.0. I tried replacing it with hadoop-client and hadoop-common but I am still getting this error for ant.filter. Can anybody please suggest which artifact to use? previous config : <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>1.0.3</version> </dependency> New Config: <dependency>

问题 We have a spark streaming application running on HDFS 2.7.3 with Yarn as the resource manager....while running the application .. these two folders /tmp/hadoop/data/nm-local-dir/filecache /tmp/hadoop/data/nm-local-dir/filecache are filling up and hence the disk ..... so from my research found that configuring these two properties in yarn-site.xml will help <property> <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name> <value>2000</value> </property> <property> <name>yarn

问题 I install spark on windows, but it failed to run showing the error below: <console>:16: error: not found: value sqlContext import sqlContext.implicits._ ^ <console>:16: error: not found: value sqlContext import sqlContext.sql ^ I tried the links below but any one of them resolves the issue: How to start Spark applications on Windows (aka Why Spark fails with NullPointerException)? Apache Spark error while start error when starting the spark shell error: not found: value sqlContext The

问题 Can you please help me out to the below scenarios. 1) While using Hadoop V2, do we use Secondary NameNode in production environment? 2) For Hadoop V2, suppose we use muliple NameNodes in active/passive connection for High Availability and when the Edits Log file is growing huge, How does the edits log gets applied to fsimage? If so, then applying the huge Edits log to Namenode would be time consuming during startup of Namenode? (We had Secondary NameNode in hadoop v1 to solve this problem)

问题 I am submitting a job to YARN (on spark 2.1.1 + kafka 0.10.2.1) which connects to a secured hbase cluster. This job, performs just fine when i am running in "local" mode (spark.master=local[*]). However, as soon as I submit the job with master as YARN (and deploy mode as client), I see the following error message - Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user I am following hortonworks recommendations for providing information to yarn cluster

问题 I have set up a multi node Hadoop Cluster. The NameNode and Secondary namenode runs on the same machine and the cluster has only one Datanode. All the nodes are configured on Amazon EC2 machines. Following are the configuration files on the master node: masters 54.68.218.192 (public IP of the master node) slaves 54.68.169.62 (public IP of the slave node) core-site.xml <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration>

问题 How to find the log file Please guide I have checked in the url of Resouce manager. But i didnt find any log file This is the complete error Query ID = hadoop_20170325120040_d54d136a-1904-4af9-8f8d-4167343db072 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Job running in-process (local Hadoop) 2017-03-25 12:00:42,954 Stage-0 map = 0%, reduce = 0% Ended Job = job_local644049657_0014 with errors Error during job, obtaining debugging

问题 I am using hive 0.13.1 and hashing combination of keys using default hive hash function. Something like select hash (date,token1,token2, parameters["a"],parameters["b"], parameters["c"]) from table1; I ran it on 150M rows. For 60% of the rows, it hashed it correctly. For the remaining rows, it gave 0. null or 1 as hash. I looked at the rows which resulted in bad hashes, I don't see anything wrong with the rows. What could be causing it? 回答1: The hash function returns 0 only when all supplied

问题 I am using hive 0.13.1 and hashing combination of keys using default hive hash function. Something like select hash (date,token1,token2, parameters["a"],parameters["b"], parameters["c"]) from table1; I ran it on 150M rows. For 60% of the rows, it hashed it correctly. For the remaining rows, it gave 0. null or 1 as hash. I looked at the rows which resulted in bad hashes, I don't see anything wrong with the rows. What could be causing it? 回答1: The hash function returns 0 only when all supplied