hdp

Simple example for adding relationships between Atlas entities?

对着背影说爱祢 提交于 2019-12-11 14:05:10
问题 What is the correct way to use the REST API to add a relationship between entities in apache atlas? Looking at the docs for the REST API, I find it difficult to tell what some of the fields mean, which are required or not (and what happens if they are not entered), or what default values should be (since the examples use what appear to be placeholder values (eg. what is provenanceType or the propagateTags field appears to expect some kind of enumeration value, but never specifies valid

Hdp, Hive, Lateral view and null: disappearing rows

点点圈 提交于 2019-12-11 06:16:41
问题 Since the upgrade from hdp 3.1.0 to 3.1.4, I have some issue in Hive I do not understand. Note that I am only using ORC transactional tables. For instance this query: with cte as ( select e.id , '{}' as json from event e ) -- select count(*) from cte select id , lv.customfield from cte lateral view outer json_tuple(cte.json, 'customfield') cv AS `customfield` It worked perfectly before the upgrade. Now, even if the CTE returns a certain number of rows, using the lateral view will just drop

Apache Spark 2.3.1 with Hive metastore 3.1.0

亡梦爱人 提交于 2019-12-07 09:11:41
问题 We have upgraded HDP cluster to 3.1.1.3.0.1.0-187 and have discovered: Hive has a new metastore location Spark can't see Hive databases In fact we see: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ... not found Could you help me understanding what has happened and how to solve this? Update: Configuration: (spark.sql.warehouse.dir,/warehouse/tablespace/external/hive/) (spark.admin.acls,) (spark.yarn.dist.files,file:///opt/folder/config.yml,file:///opt/jdk1.8.0_172

Hue 3.9 on HDP2.3.4 安装备忘

邮差的信 提交于 2019-12-06 19:40:41
安装步骤 基于CentOS,使用root账号 准备环境 使用 Ambari 安装HDP2.3.4 按 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configure_hdp_hue.html 配置HDP 重启HDP 下载 maven 及 ant 配置profile,加入几个环境变量: export JAVA_HOME=/usr/jdk64/jdk1.8.0_60 export ANT_HOME=/hadoop/program/apache-ant-1.9.6 export MAVEN_HOME=/hadoop/program/apache-maven-3.3.9 export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin:$MAVEN_HOME/bin source /etc/profile yum -y install gcc-c++ asciidoc cyrus-sasl-devel cyrus-sasl-gssapi krb5-devel libxml2-devel libxslt-devel mysql-devel openldap-devel python-devel sqlite-devel

Apache Spark 2.3.1 with Hive metastore 3.1.0

房东的猫 提交于 2019-12-05 14:41:15
We have upgraded HDP cluster to 3.1.1.3.0.1.0-187 and have discovered: Hive has a new metastore location Spark can't see Hive databases In fact we see: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ... not found Could you help me understanding what has happened and how to solve this? Update: Configuration: (spark.sql.warehouse.dir,/warehouse/tablespace/external/hive/) (spark.admin.acls,) (spark.yarn.dist.files,file:///opt/folder/config.yml,file:///opt/jdk1.8.0_172/jre/lib/security/cacerts) (spark.history.kerberos.keytab,/etc/security/keytabs/spark.service.keytab)

How to write a table to hive from spark without using the warehouse connector in HDP 3.1

旧巷老猫 提交于 2019-12-01 10:54:01
问题 when trying to use spark 2.3 on HDP 3.1 to write to a Hive table without the warehouse connector directly into hives schema using: spark-shell --driver-memory 16g --master local[3] --conf spark.hadoop.metastore.catalog.default=hive val df = Seq(1,2,3,4).toDF spark.sql("create database foo") df.write.saveAsTable("foo.my_table_01") fails with: Table foo.my_table_01 failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional but a:

ERROR 1066: Unable to open iterator for alias in Pig, Generic solution

妖精的绣舞 提交于 2019-11-25 22:20:58
问题 A very common, error message in Apache Pig is: ERROR 1066: Unable to open iterator for alias There are several questions where this error is mentioned, but none of them give a generic approach for dealing with it. Hence this question: What to do when you get an ERROR 1066: Unable to open iterator for alias ? 回答1: The message "ERROR 1066: Unable to open iterator for alias myAlias" suggests that there is something going wrong in the line where you use myAlias. However, usually you will see this