问题
this is the sqoop command which I am using to import data from SQL Server to Hive sqoop-import-all-tables --connect "jdbc:sqlserver://ip.ip.ip.ip\MIGERATIONSERVER;port=1433;username=sa;password=blablaq;database=sqlserverdb" --create-hive-table --hive-import --hive-database hivemtdb
The problem is that sqlserverdb has about 100 tables but when i issue this command it is just importing 6 or 7 random tables to hive. This behavior is really strange for me. I am unable to find where I am doing mistake.
EDIT :1
Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/10/13 13:17:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.3.0-227
16/10/13 13:17:38 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/10/13 13:17:38 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/10/13 13:17:38 INFO manager.SqlManager: Using default fetchSize of 1000
16/10/13 13:17:38 INFO tool.CodeGenTool: Beginning code generation
16/10/13 13:17:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [UserMessage] AS t WHERE 1=0
16/10/13 13:17:38 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.3.0-227/hadoop-mapreduce
Note: /tmp/sqoop-sherry/compile/c809ee201c0aec1edf2ed5a1ef4aed4c/UserMessage.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/10/13 13:17:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-sherry/compile/c809ee201c0aec1edf2ed5a1ef4aed4c/UserMessage.jar
16/10/13 13:17:39 INFO mapreduce.ImportJobBase: Beginning import of UserMessage
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/10/13 13:17:40 INFO impl.TimelineClientImpl: Timeline service address: http://machine-02-xx:8188/ws/v1/timeline/
16/10/13 13:17:40 INFO client.RMProxy: Connecting to ResourceManager at machine-02-xx/xxx.xx.xx.xx:8050
16/10/13 13:17:42 INFO db.DBInputFormat: Using read commited transaction isolation
16/10/13 13:17:42 INFO mapreduce.JobSubmitter: number of splits:1
16/10/13 13:17:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1475746531098_0317
16/10/13 13:17:43 INFO impl.YarnClientImpl: Submitted application application_1475746531098_0317
16/10/13 13:17:43 INFO mapreduce.Job: The url to track the job: http://machine-02-xx:8088/proxy/application_1475746531098_0317/
16/10/13 13:17:43 INFO mapreduce.Job: Running job: job_1475746531098_0317
16/10/13 13:17:48 INFO mapreduce.Job: Job job_1475746531098_0317 running in uber mode : false
16/10/13 13:17:48 INFO mapreduce.Job: map 0% reduce 0%
16/10/13 13:17:52 INFO mapreduce.Job: map 100% reduce 0%
16/10/13 13:17:52 INFO mapreduce.Job: Job job_1475746531098_0317 completed successfully
16/10/13 13:17:52 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=156179
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=0
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=3486
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=1743
Total vcore-seconds taken by all map tasks=1743
Total megabyte-seconds taken by all map tasks=2677248
Map-Reduce Framework
Map input records=0
Map output records=0
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=30
CPU time spent (ms)=980
Physical memory (bytes) snapshot=233308160
Virtual memory (bytes) snapshot=3031945216
Total committed heap usage (bytes)=180879360
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
16/10/13 13:17:52 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 12.6069 seconds (0 bytes/sec)
16/10/13 13:17:52 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/10/13 13:17:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM [UserMessage] AS t WHERE 1=0
16/10/13 13:17:52 WARN hive.TableDefWriter: Column SendDate had to be cast to a less precise type in Hive
16/10/13 13:17:52 INFO hive.HiveImport: Loading uploaded data into Hive
Logging initialized using configuration in jar:file:/usr/hdp/2.4.3.0-227/hive/lib/hive-common-1.2.1000.2.4.3.0-227.jar!/hive-log4j.properties
OK
Time taken: 1.286 seconds
Loading data to table sqlcmc.usermessage
Table sqlcmc.usermessage stats: [numFiles=1, totalSize=0]
OK
Time taken: 0.881 seconds
Note: /tmp/sqoop-sherry/compile/c809ee201c0aec1edf2ed5a1ef4aed4c/DadChMasConDig.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Logging initialized using configuration in jar:file:/usr/hdp/2.4.3.0-227/hive/lib/hive-common-1.2.1000.2.4.3.0-227.jar!/hive-log4j.properties
OK
回答1:
First of all import-all-tables will run import table for all the tables.
If you does not define, number of mapper in the job, Sqoop will pick by default 4 mappers. So, it needs table to have primary key or you specify --split-by column name.
If this is the case, you will see error like:
ERROR tool.ImportAllTablesTool: Error during import: No primary key could be found for table test. Please specify one with --split-by or perform a sequential import with '-m 1'.
So you can use 1 mapper which will make your import process slow.
Better way is to add --autoreset-to-one-mapper, it will import tables with primary key with the number of mappers mentioned in the command and it will automatically use 1 mapper for the tables without primary key.
Coming to your problem,
sqoop import failed for table DadChMasConDig.
I don't know why it is not logged on console.
In importing this table there could be exception like
Encountered IOException running import job: java.io.IOException: Hive does not support the SQL type for column
<somecolumn>
For example, varbinary is not supported.
If you import data only in HDFS, it should not be a problem. You can try:
sqoop-import-all-tables --connect "jdbc:sqlserver://ip.ip.ip.ip\MIGERATIONSERVER;port=1433;username=sa;password=blablaq;database=sqlserverdb"
回答2:
I had the same issue and the following worked for me. Though typically --create-hive-table and --hive-overwrite don't go together and doesn't make sense together. But no other combination worked and each time only 3 of 10 or a fraction of tables were getting imported
sqoop import-all-tables \
--connect jdbc:mysql://<mysql-url>/my_database \
--username sql_user \
--password sql_pwd \
--hive-import \
--hive-database test_hive \
--hive-overwrite \
--create-hive-table \
--warehouse-dir /apps/hive/warehouse/test_hive.db \
-m 1
来源：https://stackoverflow.com/questions/39996405/sqoop-import-all-tables-unable-to-import-all-tables