metastore

Error creating transactional connection factory during running Spark on Hive project in IDEA

由 一笑奈何 提交于 2021-01-27 04:51:34
问题 I am trying to setup a develop environment for a Spark Streaming project which requires write data into Hive. I have a cluster with 1 master, 2 slaves and 1 develop machine (coding in Intellij Idea 14). Within the spark shell, everything seems working fine and I am able to store data into default database in Hive via Spark 1.5 using DataFrame.write.insertInto("testtable") However when creating a scala project in IDEA and run it using same cluster with same setting, Error was thrown when

Error creating transactional connection factory during running Spark on Hive project in IDEA

由 …衆ロ難τιáo~ 提交于 2021-01-27 04:49:52
问题 I am trying to setup a develop environment for a Spark Streaming project which requires write data into Hive. I have a cluster with 1 master, 2 slaves and 1 develop machine (coding in Intellij Idea 14). Within the spark shell, everything seems working fine and I am able to store data into default database in Hive via Spark 1.5 using DataFrame.write.insertInto("testtable") However when creating a scala project in IDEA and run it using same cluster with same setting, Error was thrown when

Hive MetaStore的启动debug命令

由 只谈情不闲聊 提交于 2020-04-09 12:12:29
终于可以debug MetaStore了。 debug的命令如下： jdb org.apache.hadoop.hive.metastore.HiveMetaStore stop in org.apache.hadoop.hive.metastore.HiveMetaStore.main stop in org . apache . hadoop . hive . metastore . HiveMetaStore .startMetaStore stop in org . apache . hadoop . hive . metastore . HiveMetaStore . HMSHandler .方法名 stop in org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database stop in org.apache.hadoop.hive.metastore.ObjectStore.createDatabase 查询数据库 org . apache . hadoop . hive . metastore . HiveMetaStore $ HMSHandler .get_databases 创建表格 stop in org.apache.hadoop.hive.metastore

Hive, how do I retrieve all the database's tables columns

由 浪子不回头ぞ 提交于 2020-01-11 05:23:27
问题 I want to write the equivalent of this sql request in Hive : select * from information_schema.columns where table_schema='database_name' How can I access hive's metastore and retrieve all the columns of all the tables stored in a specific database? I know that we can do it by table via describe [table_name] but is there anyway to have all the columns for all the tables in a database in the same request? 回答1: If you want to have the ability to run such queries that return hive metadata, you

Unable to start hive metastore service or hive-shell after configuring mysql with hive

由 元气小坏坏 提交于 2019-12-31 04:28:08
问题 I know this question asked already but those answers are not helping in any way. I am spending more time to configure mysql with hive, Every time I encounter errors and I don't understand where it goes wrong ..... as you can see in this link, hive-site.xml configuration can be seen here in the link Here is file-structure So is the reason creating this question.... Hoping this helps me to resolve. I followed these links https://dzone.com/articles/how-configure-mysql-metastore hive-site.xml

Hive tables meta details needed

由 末鹿安然 提交于 2019-12-24 07:13:05
问题 select * from information_schema.columns; In MySQL gives me the dbname, table name and column details of a MySQL db. Can I get the same details in hive from any tables? 回答1: If you have configured your metastore in mysql .Then there are tables in the metastore database named DBS, COLUMNS_V2 which will have metadata of all hive DBs and tables . 回答2: Describe will meet your requirement. hive -e "desc formatted tablename" On above output you can use grep like below hive -e "desc formatted

Hive 1.2 Metastore Service doesn't start after configuring it to S3 storage instead HDFS

由 99封情书 提交于 2019-12-24 04:33:07
问题 I have an Apache Spark Cluster(2.2.0) in standalone mode. Till now was running using HDFS to store the parquet files. I'm using the Hive Metastore Service of Apache Hive 1.2 to access, using the Thriftserver, Spark over JDBC. Now I want to use S3 Object Storage instead HDFS. I have added the following configuration to my hive-site.xml: <property> <name>fs.s3a.access.key</name> <value>access_key</value> <description>Profitbricks Access Key</description> </property> <property> <name>fs.s3a

Can we predict the order of the results of a Hive SELECT * query?

由 六眼飞鱼酱① 提交于 2019-12-13 18:37:06
问题 Is it possible that the order of the results of a SELECT * query (no ORDER BY) is always the same provided that the same DBMS is used as Metastore? So, as long as MySQL is used as Metastore, the order of the results for a SELECT *; query will always be the same. If Postgres is used, the order will be always the same on the same data, but different from when MySQL is used. I am talking about the same data. Maybe it all boils down to the question of what is the default order of results and why

Google cloud dataproc failing to create new cluster with initialization scripts

由 ε祈祈猫儿з 提交于 2019-12-13 07:37:30
问题 I am using the below command to create data proc cluster: gcloud dataproc clusters create informetis-dev --initialization-actions “gs://dataproc-initialization-actions/jupyter/jupyter.sh,gs://dataproc-initialization-actions/cloud-sql-proxy/cloud-sql-proxy.sh,gs://dataproc-initialization-actions/hue/hue.sh,gs://dataproc-initialization-actions/ipython-notebook/ipython.sh,gs://dataproc-initialization-actions/tez/tez.sh,gs://dataproc-initialization-actions/oozie/oozie.sh,gs://dataproc

grant permissions in hive does not work on hdp2.2

由 你说的曾经没有我的故事 提交于 2019-12-11 04:19:18
问题 I'm experimenting with HDP2.2 cluster with Ambari setup on CentOS 6.5 and I have problems with running Hive GRANT queries. For example, a query grant select on Tbl1 to user root; gives me an exception that looks like that FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Failed to retrieve roles for hdfs: Metastore Authorization api invocation for remote metastore is disabled in this configuration. What's going on here and could you explain the meaning of
订阅 metastore