sqoop2

How to change sqoop metastore?

血红的双手。 提交于 2019-12-17 19:05:04
问题 I am using sqoop 1.4.2 version. I am trying to change the sqoop metastore from default hsqldb to mysql. I have configured following properties in sqoop-site.xml file. <property> <name>sqoop.metastore.client.enable.autoconnect</name> <value>false</value> <description>If true, Sqoop will connect to a local metastore for job management when no other metastore arguments are provided. </description> </property> <property> <name>sqoop.metastore.client.autoconnect.url</name> <value>jdbc:mysql://ip

sqoop exec job in oozie is not working

南楼画角 提交于 2019-12-13 07:42:03
问题 I am running a 3 node HDP 2.2 cluster. Oozie version is 4.1.0.2.2 and Sqoop version is 1.4.5.2.2. I am using Sqoop job to do incremental imports from RDBMS into HDFS as shown below, sqoop job –create JOB1 –meta-connect “jdbc:hsqldb:hsql://ip-address:16000/sqoop” — import –connect jdbc:oracle:thin:@ip-address:db –username db_user –password-file hdfs://ip-address:8020/user/oozie/.password_sqoop –table TABLE1 –target-dir /user/incremental/ –incremental lastmodified –check-column LAST_UPDATED

Which Maven version I need to build Sqoop2

时间秒杀一切 提交于 2019-12-13 04:42:16
问题 Please help to build Sqoop2 from source. I use Maven 3.2.1 on CentOS 6.5. When I try to compile Sqoop2 with command: mvn compile I get exception resulting from: missing org.apache.sqoop:sqoop-core:test-jar:tests:2.0.0-SNAPSHOT When I try to build Sqoop2 with command: mvn package -Pbinary I get exception resulting from: Missing org/sonatype/aether/graph/DependencyFilter Any ideas? 回答1: The same issue: java.lang.NoClassDefFoundError: org/sonatype/aether/graph/DependencyFilter Caused by: java

sqoop to import data to hive

≡放荡痞女 提交于 2019-12-12 01:22:15
问题 i am trying to import data to hive table using sqoop2. I am using --hive-import but it is not working Code: sqoop import --connect jdbc:sqlserver://192.168.x.xxx:11xx --username user --password user --table xxxx.NOTIFICATION --hive-import Error: ERROR manager.SqlManager: Error executing statement: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'XXXX.NOTIFICATION'. What am I doing wrong? 回答1: Below observations are based on Sqoop 1.4.6 you are using . (dot) in your table

SQOOP incremental import: how it handles the data when a row is deleted from the database?

你离开我真会死。 提交于 2019-12-11 03:54:31
问题 Suppose I have an employee table with column ( emp_id, emp_name, emp_age , emp_update_ts ), updat_ts field is auto updated to current timestamp every time if there is an update on the table. now my question is : When I update/insert the row in the table and run incremental sqoop import with lastmodified it will make my database and the hdfs in sync but "what if the data is deleted from the table and then I run the sqoop incremental with lastmodified option ? will it also take care of deleting

SQOOP Not able to import table

笑着哭i 提交于 2019-12-10 19:52:43
问题 I am running below command on sqoop sqoop import --connect jdbc:mysql://localhost/hadoopguide --table widgets my version of sqoop : Sqoop 1.4.4.2.0.6.1-101 Hadoop -- Hadoop 2.2.0.2.0.6.0-101 Both taken from hortonworks distribution. all the paths like HADOOP_HOME, HCAT_HOME, SQOOP_HOME are set properly. I am able to get list of databases, list of tables from mysql database by running list-database, list-tables commands in sqoop. Even able to get data from --query 'select * from widgets'; but

What is --direct mode in sqoop?

爱⌒轻易说出口 提交于 2019-12-05 23:35:43
问题 As per my understanding sqoop is used to import or export table/data from the Database to HDFS or Hive or HBASE. And we can directly import a single table or list of tables. Internally mapreduce program (i think only map task) will run. My doubt is what is sqoop direct and what when to go with sqoop direct option? 回答1: Just read the Sqoop documentation! General principles are located here for imports and there for exports Some databases can perform imports in a more high-performance fashion

Sqoop - Binding to YARN queues

泄露秘密 提交于 2019-12-05 06:30:52
So with mapreduce v2 you can use binding to certain YARN queues to manage resources and prioritization. Basically by using "hadoop jar /xyz.jar -D mapreduce.job.queuename=QUEUE1 /input /output" which works perfectly. How can integrate Yarn queue binding with Sqoop when you run a sqoop query? ie. sqoop import \ --connect 'jdbc://server' \ --target-dir \ and what ? Use the same method for Sqoop as well i.e sqoop import -Dmapreduce.job.queuename=NameOfTheQueue\ --connect 'jdbc://server' \ --target-dir \ Note that, the mapreduce.job.queuename parameter should be used after the import but before

What is --direct mode in sqoop?

旧巷老猫 提交于 2019-12-04 06:16:36
As per my understanding sqoop is used to import or export table/data from the Database to HDFS or Hive or HBASE. And we can directly import a single table or list of tables. Internally mapreduce program (i think only map task) will run. My doubt is what is sqoop direct and what when to go with sqoop direct option? Just read the Sqoop documentation! General principles are located here for imports and there for exports Some databases can perform imports in a more high-performance fashion by using database-specific data movement tools (...) Some databases provides a direct mode for exports as

How to change sqoop metastore?

折月煮酒 提交于 2019-11-28 09:31:10
I am using sqoop 1.4.2 version. I am trying to change the sqoop metastore from default hsqldb to mysql. I have configured following properties in sqoop-site.xml file. <property> <name>sqoop.metastore.client.enable.autoconnect</name> <value>false</value> <description>If true, Sqoop will connect to a local metastore for job management when no other metastore arguments are provided. </description> </property> <property> <name>sqoop.metastore.client.autoconnect.url</name> <value>jdbc:mysql://ip:3206/sqoop?createDatabaseIfNotExist=true</value> </property> <property> <name>sqoop.metastore.client