sqoop

ERROR tool.ImportTool: Import failed: java.io.IOException: Could not load jar /tmp/sqoop-root/

由 China☆狼群 提交于 2020-01-16 14:37:07
问题 when I run the following sqoop merge import to update an existing Hive table that I already have: sudo sqoop import \ --connect 'jdbc:sqlserver://1.1.1.1\test_server;database=Training' \ --username Training_user --password Training_user \ --table BigDataTest -m 1 \ --check-column lastmodified \ --merge-key id \ --incremental lastmodified \ --compression-codec=snappy \ --as-parquetfile \ --target-dir /user/hive/warehouse \ --hive-table bigDataTest \ --last-value '2019-05-06 15:07:49.917' I get

大数据架构师从入门到精通，该具备怎么样的知识体系？

由 不羁的心 提交于 2020-01-14 14:19:27
经常有初学者在博客和QQ问我，自己想往大数据方向发展，该学哪些技术，学习路线是什么样的，觉得大数据很火，就业很好，薪资很高。如果自己很迷茫，为了这些原因想往大数据方向发展，也可以，那么我就想问一下，你的专业是什么，对于计算机/软件，你的兴趣是什么？是计算机专业，对操作系统、硬件、网络、服务器感兴趣？是软件专业，对软件开发、编程、写代码感兴趣？还是数学、统计学专业，对数据和数字特别感兴趣。 其实这就是想告诉你的大数据的三个发展方向，平台搭建/优化/运维/监控、大数据开发/ 设计/ 架构、数据分析/挖掘。请不要问我哪个容易，哪个前景好，哪个钱多。 先扯一下大数据的4V特征： 数据量大 ，TB->PB 数据类型繁多 ，结构化、非结构化文本、日志、视频、图片、地理位置等； 商业价值高 ，但是这种价值需要在海量数据之上，通过数据分析与机器学习更快速的挖掘出来； 处理时效性高 ，海量数据的处理需求不再局限在离线计算当中。 现如今，正式为了应对大数据的这几个特点，开源的大数据框架越来越多，越来越强，先列举一些常见的： 文件存储： Hadoop HDFS、Tachyon、KFS 离线计算： Hadoop MapReduce、Spark 流式、实时计算： Storm、Spark Streaming、S4、Heron K-V、NOSQL数据库： HBase、Redis、MongoDB 资源管理：

sqoop integration with hadoop throw ClassNotFoundException

由 ぐ巨炮叔叔 提交于 2020-01-14 06:20:50
问题 I am new in word of hadoop and sqoop. I installed hadoop 2.7.3 (pseudo mode) and its working fine on my system. I want integration with sqoop. I am using sqoop sqoop-1.99.7-bin-hadoop200. 1) I extract tar file and move extracted content into /usr/local/sqoop 2) Set Sqoop path into .bashrc file. 3) go to /usr/local/sqoop/server/lib/sqoop.sh server start and get following error message. hadoop_usr@sawai-Lenovo-G580:/usr/local/sqoop/server/lib$ sqoop.sh server start Setting conf dir: /usr/local

Moving HDFS data into MongoDB

由 我只是一个虾纸丫 提交于 2020-01-14 06:01:29
问题 I am trying to move HDFS data into MongoDB. I know how to export data into mysql by using sqoop. I dont think I can use sqoop for MongoDb. I need help understanding how to do that. 回答1: The basic problem is that mongo stores its data in BSON format (binary JSON), while you hdfs data may have different formats (txt, sequence, avro). The easiest thing to do would be to use pig to load your results using this driver: https://github.com/mongodb/mongo-hadoop/tree/master/pig into mongo db. You'll

Moving HDFS data into MongoDB

由 一笑奈何 提交于 2020-01-14 06:01:08
问题 I am trying to move HDFS data into MongoDB. I know how to export data into mysql by using sqoop. I dont think I can use sqoop for MongoDb. I need help understanding how to do that. 回答1: The basic problem is that mongo stores its data in BSON format (binary JSON), while you hdfs data may have different formats (txt, sequence, avro). The easiest thing to do would be to use pig to load your results using this driver: https://github.com/mongodb/mongo-hadoop/tree/master/pig into mongo db. You'll

How to support column names with spaces using sqoop import?

由 余生长醉 提交于 2020-01-14 03:48:07
问题 We have an MSSQL DB set-up with column names of "Column 0" and "Column 1": note the space. If I run the following command, it errors: sqoop import --driver net.sourceforge.jtds.jdbc.Driver --connect jdbc:jtds:sqlserver://somemssqldb.com/OurDB --table dbo.OurTableName --username username --password ourPassword --columns "Column 0" --target-dir s3://our-s3-bucket/9/data/1262/141893327230246 -m 1 Stack trace reports: Error: java.io.IOException: SQLException in nextKeyValue Caused by: java.sql

Sqoop整合HBase

由 筅森魡賤 提交于 2020-01-14 02:34:30
sqoop是一个数据导入导出的工具，可以将关系型数据库当中的数据导入到大数据平台来，也可以将大数据平台当中的数据导入到关系型数据库当中去 我们也可以通过sqoop导入数据到hbase或者从hbase当中导出数据 需求一 ： 将 mysql表当中的数据导入到 HB ase当中来 第一步 ： 修改 sqoop 配置 文件 sqoop导入导出HBase的数据，需要修改sqoop的配置文件sqoop-env.sh cd /export/servers/sqoop-1.4.6-cdh5.14.0/conf vim sqoop-env.sh #Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/export/servers/hadoop-2.6.0-cdh5.14.0 #Set path to where hadoop-*-core.jar is available export HADOOP_MAPRED_HOME=/export/servers/hadoop-2.6.0-cdh5.14.0 #set the path to where bin/hbase is available export HBASE_HOME=/export/servers/hbase-1.2.0-cdh5.14.0 #Set

Sqoop Free-Form Query Causing Unrecognized Arguments in Hue/Oozie

由 不羁岁月 提交于 2020-01-13 19:49:47
问题 I am attempting to run a sqoop command with a free-form query, because I need to perform an aggregation. It's being submitted via the Hue interface, as an Oozie workflow. The following is a scaled-down version of the command and query. When the command is processed, the "--query" statement (enclosed in quotes) results in each portion of the query to be interpreted as unrecognized arguments, as shown in the error following the command. In addition, the target directory is being misinterpreted.

Sqoop Free-Form Query Causing Unrecognized Arguments in Hue/Oozie

由 こ雲淡風輕ζ 提交于 2020-01-13 19:49:31
问题 I am attempting to run a sqoop command with a free-form query, because I need to perform an aggregation. It's being submitted via the Hue interface, as an Oozie workflow. The following is a scaled-down version of the command and query. When the command is processed, the "--query" statement (enclosed in quotes) results in each portion of the query to be interpreted as unrecognized arguments, as shown in the error following the command. In addition, the target directory is being misinterpreted.

Apache Sqoop导入&导出

由 ∥☆過路亽．° 提交于 2020-01-11 23:18:11
RDBMS->HDFS 全表导入 sqoop import \ --driver com.mysql.jdbc.Driver \ --connect jdbc:mysql://CentOS:3306/test?characterEncoding = UTF-8 \ --username root \ --password root \ --table t_user \ --num-mappers 4 \ --fields-terminated-by '\t' \ --target-dir /mysql/test/t_user \ --delete-target-dir 参数 含义 –-connect 连接的数据库地址 -–username 连接的数据库的用户名 –-password 连接的数据库的密码 –-table 想要导出数据的表 –target-dir 要导入到hdfs中的目录（如果不指定，默认存储在“/user/用户名/导入的表名” 目录下） -–delete-target-dir 表示如果在hdfs中有该目录，则先删除，然后再导入数据到该目录下 –num-mappers 表示设置的maptask个数，默认为4个，决定最终在hdfs中生成的文件个数（将table中的数据分成几个文件分别存储） –fields-terminated-by 指定字段的分割符号 字段导入
订阅 sqoop