hortonworks-data-platform

I've been strugling with distcp for several days and I swear I have googled enough. Here is my use-case: USE CASE I have a main folder in a certain location say /hdfs/root , with a lot of subdirs (deepness is not fixed) and files. Volume: 200,000 files ~= 30 GO I need to copy only a subset for a client, /hdfs/root in another location, say /hdfs/dest This subset is defined by a list of absolute path that can be updated over time. Volume: 50,000 files ~= 5 GO You understand that I can't use a simple hdfs dfs -cp /hdfs/root /hdfs dest because it is not optimized, it will take every files, and it

i have a problem in setting hadoop file permissions in hortonworks and cloudera. My requirement is: 1. create a new user with new group 2. create user directory in hdfs ( ex. /user/myuser ) 3. Now this folder ( in this case /user/myuser ) must be accessible to only user and its group but not other users and other groups. Following commands are used by me. ( in centos 6) 1.create group >>> groupadd mygroup 2. create new user who belongs to new group >>>> useradd -g mygroup myuser 3. create user directory in hdfs >>> hadoop fs -mkdir /user/myuser 4. changing ownership of the folder >>> hadoop fs

We have developed a Spring Boot Application to query data from Hbase on HDP 2.5. The application works fine when we run it on our local windows machines as it is able to find the jaas.conf & krb5.conf file which have been kept under resources folder of the application and we have provided them as system properties in our code as e.g. System.setProperties("javax.security.auth.useSubjectCredsOnly","false"); System.setProperty("java.security.krb5.conf","D:\\host\\workspace\\projectname\\src\\main\\resources\\krb5.conf"); However when we deploy in PCF it is unable to read those file which are set

I am using Hortonworks Sandbox 2.0 which contains the following version of Hbase and Hive Component Version ------------------------ Apache Hadoop 2.2.0 Apache Hive 0.12.0 Apache HBase 0.96.0 Apache ZooKeeper 3.4.5 ...and I am trying to register my hbase table into hive using the following query CREATE TABLE IF NOT EXISTS Document_Table_Hive (key STRING, author STRING, category STRING) STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’ WITH SERDEPROPERTIES (‘hbase.columns.mapping’ = ‘:key,metadata:author,categories:category’) TBLPROPERTIES (‘hbase.table.name’ = ‘Document’); This does

I'm using hortonworks 2.1 with hbase and hive. I want to create a hbase table from tsv file. The file is here: id c1 c2 row1 1 22 row2 e1 42 row3 g1 f2 row4 f1 c2 row5 d1 c2 row6 c1 42 row7 e1 c2 row8 c1 c2 row9 c1 c2 row10 c1 22 The command I used is: bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns= HBASE_ROW_KEY,d:c1,d:c2 hbaseTable /tmp/testTSV.tsv But I got this error: SyntaxError: (hbase):5: syntax error, unexpected tSYMBEG this is the error: hbase(main):007:0> ImportTsv -Dimporttsv.columns= HBASE_ROW_KEY,d:c1,d:c2 hbaseTable/tmp/testTSV.tsv SyntaxError: (hbase)

Getting error while sending message to kafka topic in kerberosed enviornment. We have cluster on hdp 2.3 I followed this http://henning.kropponline.de/2016/02/21/secure-kafka-java-producer-with-kerberos/ But for sending messages, I have to do kinit explicitly first, then only I am able to send message to kafka topic. I tried to do knit through java class but that also doesn't work. PFB code: package com.ct.test.kafka; import java.util.Date; import java.util.Properties; import java.util.Random; import kafka.javaapi.producer.Producer; import kafka.producer.KeyedMessage; import kafka.producer