hortonworks-data-platform

Before any Answer and Comments. I tried several option I found in Stackoverflow but end with a failure. Following are those links - How can I execute Sqoop in Java? How to use Sqoop in Java Program? How to import table from MySQL to Hive using Java? How to load SQL data into the Hortonworks? I tried it in Horton Sandbox through command line and succeded. sqoop import --connect jdbc:mysql://192.168.56.101:3316/database_name --username=user --password=pwd --table table_name --hive-import -m 1 -- --schema default Where 192.168.56.101 is for Windows and 192.168.56.102 for Horton Sandbox 2.6. Now I

If we can change the data block size in Hadoop please let me know how to do that. Is it advantageous to change the block size, If yes, then let me know Why and how? If no, then let me know why and how? There seems to be much confusion about this topic and also wrong advise going around. To lift the confusion it helps to think about how HDFS is actually implemented: HDFS is an abstraction over distributed disk-based file systems. So the words "block" and "blocksize" have a different meaning than generally understood. For HDFS a "file" is just a collection of blocks, each "block" in return is

I'm new to Spark on YARN and don't understand the relation between the YARN Containers and the Spark Executors . I tried out the following configuration, based on the results of the yarn-utils.py script, that can be used to find optimal cluster configuration. The Hadoop cluster (HDP 2.4) I'm working on: 1 Master Node: CPU: 2 CPUs with 6 cores each = 12 cores RAM: 64 GB SSD: 2 x 512 GB 5 Slave Nodes: CPU: 2 CPUs with 6 cores each = 12 cores RAM: 64 GB HDD: 4 x 3 TB = 12 TB HBase is installed (this is one of the parameters for the script below) So I ran python yarn-utils.py -c 12 -m 64 -d 4 -k

I have a Spark (version 1.4.1) application on HDP 2.3. It works fine when running it in YARN-Client mode. However, when running it on YARN-Cluster mode none of my Hive tables can be found by the application. I submit the application like so: ./bin/spark-submit --class com.myCompany.Main --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 10g --executor-cores 1 --jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar /home/spark/apps/YarnClusterTest.jar --files /etc/hive/conf/hive-site.xml Here's a excerpt from the logs: 5

This is a follow up question to this question where I ask what the Hiveserver 2 thrift java client API is. This question should be able to stand along without that background if you don't need any more context. Unable to find any documentation on how to use the hiverserver2 thrift api, I put this together. The best reference I could find was the Apache JDBC implementation . TSocket transport = new TSocket("hive.example.com", 10002); transport.setTimeout(999999999); TBinaryProtocol protocol = new TBinaryProtocol(transport); TCLIService.Client client = new TCLIService.Client(protocol); transport