hive-metastore

We have upgraded HDP cluster to 3.1.1.3.0.1.0-187 and have discovered: Hive has a new metastore location Spark can't see Hive databases In fact we see: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ... not found Could you help me understanding what has happened and how to solve this? Update: Configuration: (spark.sql.warehouse.dir,/warehouse/tablespace/external/hive/) (spark.admin.acls,) (spark.yarn.dist.files,file:///opt/folder/config.yml,file:///opt/jdk1.8.0_172/jre/lib/security/cacerts) (spark.history.kerberos.keytab,/etc/security/keytabs/spark.service.keytab)

I'm working in an environment where I have an S3 service being used as a data lake, but not AWS Athena. I'm trying to setup Presto to be able to query the data in S3 and I know I need the define the data structure as Hive tables through the Hive Metastore service. I'm deploying each component in Docker, so I'd like to keep the container size as minimal as possible. What components from Hive do I need to be able to just run the Metastore service? I don't really actually care about running Hive, just the Metastore. Can I trim down what's needed, or is there already a pre-configured package just

I am having an AWS EMR cluster (v5.11.1) with Spark(v2.2.1) and trying to use AWS Glue Data Catalog as its metastore. As per guidelines provided in official AWS documentation (reference link below), I have followed the steps but I am facing some discrepancy with regards to accessing the Glue Catalog DB/Tables. Both EMR Cluster & AWS Glue are in the same account and appropriate IAM permissions have been provided. AWS Documentation : https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html Observations: - Using spark-shell (From EMR Master Node): Works . Able to access Glue DB

I am trying to understand hive in terms of architecture, and I am referring to Tom White's book on Hadoop. I came across the following terms in regards to hive: Hive Services , hiveserver2 , metastore among others. Referring to below diagrams from the Book (Hadoop: The definitive Guide). Hive Architecture: MetaStore configuration: Hive Architecture which shows what "Driver" is: I am not able to understand the following: 1) What is Hive Services in Hive architecture diagram? Is it same when we say hiveserver2 ? 2) What is Driver in Hive architecture diagram? 3) What is MetaStore (I am NOT

This is an extension of a previous question I asked: Is it possible to change partition metadata in HIVE? We are exploring the idea of changing the metadata on the table as opposed to performing a CAST operation on the data in SELECT statements. Changing the metadata in the MySQL metastore is easy enough. But, is it possible to have that metadata change applied to a column that is on a partitioned table (they are daily)? Note: the column itself is not the partitioning column. It is a simple ID field that is being changed from STRING to BIGINT. Otherwise, we might be stuck with current and

I have configured my Hive as given on link: http://www.youtube.com/watch?v=Dqo1ahdBK_A , but I am getting the following error while creating a table in Hive. I am using hadoop-1.2.1 and hive-0.12.0. hive> create table employee(emp_id int,name string,salary double); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient Looks like problem with your metastore. If you are using the default hive metastore embedded derby. Lock file would be there in case of abnormal