external-tables | 易学教程

Confusion with the external tables in hive

阅读更多关于 Confusion with the external tables in hive

问题 I have created the hive external table using below command: use hive2; create external table depTable (depId int comment 'This is the unique id for each dep', depName string,location string) comment 'department table' row format delimited fields terminated by "," stored as textfile location '/dataDir/'; Now, when I view the HDFS I can see the db but there is no depTable inside the warehouse. [cloudera@quickstart ~]$ hadoop fs -ls /user/hive/warehouse/hive2.db [cloudera@quickstart ~]$ Above

Oracle external table date field - works in one DB and not in another

阅读更多关于 Oracle external table date field - works in one DB and not in another

问题 Here's a crazy one: the same external table definition works fine in one database, but fails in another. Not schema - database. Two databases, both on the same OS, different servers. In addition, it's failing on the 2nd date field, though both are defined the same. The NLS settings are the same on both servers, thought I thought the date mask should override that anyway. Here's the definition: -- access parameters -- http://docs.oracle.com/cd/E11882_01/server.112/e16536/et_params.htm CREATE

Automated List of Hive External tables

阅读更多关于 Automated List of Hive External tables

问题 I have to create an automated process to list all external tables in Hive and do a record count on those tables. I should do this as a daily job. I tried this by hard coding all the external table names, but this is not accepted as the tables keep on changing once in a month. I have gone through different approaches like [show tables] and executing query in metastore DB. But these will not help me in automating the process. Is there a better approach to implement this in Hive. 回答1: Something

Create External table in Azure databricks

阅读更多关于 Create External table in Azure databricks

问题 I am new to azure databricks and trying to create an external table, pointing to Azure Data Lake Storage (ADLS) Gen-2 location. From databricks notebook i have tried to set the spark configuration for ADLS access. Still i am unable to execute the DDL created. Note : One solution working for me is mounting the ADLS account to cluster and then use the mount location in external table's DDL. But i needed to check if it is possible to create a external table DDL with ADLS path without mount

How do “Fixed-length records” and “Fixed-length fields” increases database performance?

阅读更多关于 How do “Fixed-length records” and “Fixed-length fields” increases database performance?

问题 Could anyone please explain the below two statements w.r.t the Oracle external table performance improvement with the ORACLE_LOADER access driver: Fixed-length records are processed faster than records terminated by a string. Fixed-length fields are processed faster than delimited fields. Explanation with code might help me to understand the concept in depth. here is the two syntax(s): Fixed field length create table ext_table_fixed ( field_1 char(4), field_2 char(30) ) organization external

Does DROP PARTITION delete data from external table in HIVE?

阅读更多关于 Does DROP PARTITION delete data from external table in HIVE?

问题 An external table in HIVE is partitioned on year, month and day. So does the following query delete data from external table for the specific partitioned referenced in this query?:- ALTER TABLE MyTable DROP IF EXISTS PARTITION(year=2016,month=7,day=11); 回答1: Partitioning scheme is not data. Partitioning scheme is part of table DDL stored in metadata (simply saying: partition key value + location where the data-files are being stored). Data itself are stored in files in the partition location

Create External table in Azure databricks

阅读更多关于 Create External table in Azure databricks

I am new to azure databricks and trying to create an external table, pointing to Azure Data Lake Storage (ADLS) Gen-2 location. From databricks notebook i have tried to set the spark configuration for ADLS access. Still i am unable to execute the DDL created. Note : One solution working for me is mounting the ADLS account to cluster and then use the mount location in external table's DDL. But i needed to check if it is possible to create a external table DDL with ADLS path without mount location. # Using Principal credentials spark.conf.set("dfs.azure.account.auth.type", "OAuth") spark.conf

How do “Fixed-length records” and “Fixed-length fields” increases database performance?

阅读更多关于 How do “Fixed-length records” and “Fixed-length fields” increases database performance?

Could anyone please explain the below two statements w.r.t the Oracle external table performance improvement with the ORACLE_LOADER access driver: Fixed-length records are processed faster than records terminated by a string. Fixed-length fields are processed faster than delimited fields. Explanation with code might help me to understand the concept in depth. here is the two syntax(s): Fixed field length create table ext_table_fixed ( field_1 char(4), field_2 char(30) ) organization external ( type oracle_loader default directory ext_dir access parameters ( records delimited by newline fields

How to truncate a partitioned external table in hive?

阅读更多关于 How to truncate a partitioned external table in hive?

I'm planning to truncate the hive external table which has one partition. So, I have used the following command to truncate the table : hive> truncate table abc; But, it is throwing me an error stating : Cannot truncate non-managed table abc. Can anyone please suggest me out regarding the same ... Make your table MANAGED first: ALTER TABLE abc SET TBLPROPERTIES('EXTERNAL'='FALSE'); Then truncate : truncate table abc; And finally you can make it external again: ALTER TABLE abc SET TBLPROPERTIES('EXTERNAL'='TRUE'); 来源： https://stackoverflow.com/questions/53257144/how-to-truncate-a-partitioned

Getting an external table's location and file name?

阅读更多关于 Getting an external table's location and file name?

I'm processing multiple files as part of an external table. Is there any way I can get the file name being processed in external tables and place it in the table? At the moment the only solution I can find is appending the file name to every record in the flat file which isn't ideal from an efficiency standpoint and involves modifying the original data. Surely external tables know what file is being processed at any time? I'm not aware of any way to capture the file name within the access parameters. As a workaround, rather than modifying the original files you could use a preprocessor to